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An evaluation was made of the Effective Schools 
Program (ESP) conducted by the Mid-Continent Regional Educational 
Laboratory (NcREL) for the 1984-85 academic year. The ESP is a 
research-based program designed to foster self-sustained improvement 
efforts at the school building level, through systematic long term 
staff development including carefully planned training sessions 
designed to develop building level leadership teams consisting of 
teachers and administrators. The program is based on recent research 
related to effective schooling practices, educational leadership, 
organizational change, staff development, curriculum, and assessment. 
This report of the program evaluation consists of four sections 
containing: (1) a description of the ESP and a review of the research 
upon which the program is based; (2) a description of the evaluation 
design; (3) data results; and (4) a fiuamary discuss ion « Among the 
results of the program evaluation was the major finding that ESP did 
have the hypothesized effects for all stages and levels of the 
program. More detail is given regarding effects on student 
achievement, building policies and practices, classroom behavior, 
student behavior, and participant knowledge of the school 
effectiveness literature. Seven appendixes , comprising over half the 
document, consist of various study instruments and data collection 
forms, including numerous charts and iigures. A nine-page 
bibliography is also provided. (CB) 
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INTRODUCTION 



Mid-continent Regional Educational Laboratory's Effective Schools 
Program (McREL-ESP) is a research-based program designed to foster 
self-sustained improvement efforts at the school building level • This 
document reports the results of the efforts to evaluate McREL's Effective 
Schools Program during the 1984-85 academic year. 

The report includes five sections. Section I contains a description of 
the ESP and a review of the research upon which the program is based. 
Section II describes the evaluation design. The results of the data 
are reported in Sections III and and discussed in Section IV. Instruments 
and data collection forms are reported in the Appendices. 

The 1984-85 McREL ESP evaluation effort utilized all available data to 
document the programs effectiveness. Due to contractual agreements it is 
difficult to impose the expense of additional data collection upon school 
districts for the purpose of program evaluation. Therefore, McREL has had 
limited resources with which to perform an evaluation. Consequently, much 
data is qualitative in nature and in no case has McREL been able to utilize 
control group evaluation designs. However, given the breadth of data 
collected for the evaluation, McREL has been able to make some highly 
defensable inferences regarding the effects of the ESP on participants and 
students . 
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SECTION 1 
PROGRAM DESCRIPTION 



The Effective Schools Program of the Mid-continent Regional Educational 
Laboratory (McREL-ESP) is a systematic long-term staff-development program 
which includes carefully planned training sessions designed to develop 
building-level leadership teams consisting of teachers and administrators. 
The program is based on recent research related to effective schooling 
practices. That is, McREL's ESP combines the research on effective teaching 
and instruction with the relevant literature on education leadership, 
organization change, staff development, curriculum and assessment. 

This section contains: 1) a brief review of the research upon which the 
ESP is based; 2) a description of the intended goals of the program; 3) a 
description of how the program is delivered and 4) a description of the 
program activities. 

REVIEW OF RESEARCH 

Building Level Research . School level research on academic effectiveness 
continues to receive considerable attention. The phenomena of building level 
impact on student achievement have been carefully investigated for a number 
of years. 

Recent work (Mullen & Summers, 1983; Muname, 1980; Rutter et al., 1979) 
has confirmed some of the earlier building-level findings of Coleman et al . 
(1966), Jencks (1972), and others. Static, quantitative variables such as 
building age, amount of equipment, per capita student expenditures, and years 
of teacher professional training appear to have a negligible effect on 
student achievement. In contrast, school effectiveness, as measured by 
student performance, is affected by contextual aspects of the building, and 
the content and focus of building activities. For example, the work of Baron 
and Shoemaker (1982), Rutter et al , (1979), United States Department of 
Health, Education, and Welfare (1978), and Edmonds (1982) suggests that 
student achievement is influenced positively by an overall safe, orderly, 
clean and predictable school environment. Other elements of a school's 
climate have been shown to be important as well. These elements include 
strong academic emphasis within the building, the amount of time scheduled 
for instruction, a consistently applied system for monitoring student 
performance, (e.g., Edmonds, 1982), and a reward system for student academic 
performance, (e.g. Rutter et al . , 1979). Additionally, cooperation among a 
school's staff, positive attitudes regarding the school, and high 
expectations for students and teacher performance have been described as 
characteristics of schools where students achieve at higher levels (Armor et 
al., 1976; Berliner, 1979; US DHEW, 1978; Westbrook, 1982). 

The role and behavior of the principal is another building level 
variable which has received significant attention in school-focused research. 
One of the most consistent findings in the research is that strong leadership 
exists in schools that foster high student performance (Armor et al . . 1976; 
Averech et al . , 1974; Baron & Shoemaker, 1982; Berman & McLaughlin, 1977; 
Brookover & Leaotte, 1979; Brophy & Good, 1970; California State Department 
of Education, 1980; Glenn, 19Sl;'Levine & Stark, 1982; Mayeske et al . , 1972; 
New York State Department or Education, 1974; Purkey & Smith, 1982; Trisman 
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et al., 1976; Venezky & Winfield, 1972; Weber, 1971). The importance of the 
principal's role in influencing school climate (Berman & McLaughlin, 1978; 
Young, 1980) and the other school characteristics mentioned above has been 
veil established. 

Classroom Research * The body of research which focuses on the classroom can 
be subdivided into several different areas. The most important of these 
includes teacher behaviors related to classroom organization, classroom 
management, instruction, attitudes and expectations. In addition, student 
behaviors have received considerable attention, along with such 
learning-related issues as time-on-task, engagement and success rates, and 
"Academic Learning Time," as well as learning styles (Berliner, 1979; Bloom, 
1974; Fisher et al . , 1978). 

Similar to the findings at the building-level, the environment of the 
classroom has been shown to affect student learning. For example, classrooms 
with a climate which stimulates positive student attitudes toward learning, 
heightens students* confidence, and communicates a sense of student control 
over personal destiny can facilitate learning (Austin, 1579; California State 
Department of Education, 1980; Coleman et al., 1966; Fisher et al., 1978; 
Hamilton, 1982; Rutter et al., 1979; Squires, 1980; US DHEW, 1978). 

The teacher's classroom organization and management strategies 
contribute greatly to such a contextual framework. Similar to findings at 
the building level, a sense of order and predictability is important in the 
classroom. This includes consistently enforced rules and procedures which 
are clearly defined and communicated (Evertson, 1980; Rutter et al., 1979). 
The amount of academic emphasis in classroom activities, (e.g., the relative 
amount of time spent on learning versus management tasks) is also related to 
student learning (Berliner, 1979; Fisher et al., 1978). 

Classroom discipline and management of student behavior are also 
important factors. Teachers who are effective in stimulating student 
achievement are characterized as applying disciplinary procedures which are 
clearly defined in terms of acceptable or unacceptable behaviors. These 
teachers do not denigrate misbehaving students and make every attempt not to 
interrupt the learning process (Emmer and Evertson, 1979). Student learning 
is also influenced by strategies for beginning the school year, such as the 
teacher's choice of activities and topics during the first several days of 
school (Emmer and Evertson, 1979). 

Another area of the classroom-focused research concentrates on teachers' 
organization and instruction and their relationship to student achievement. 
Lessons that are well-structured and organized and include the use of 
presentations, discussions, frequent reviews, academic feedback, and certain 
patterns of student groupings, are characteristics of effective teachers 
(Berliner, 1979; Good, 1981; Good and Grouws , 1979; Medley, 1977; Rosenshine, 
1979). Behavioral techniques and teacher actions are also direcily related 
to student achievement. These include high levels of teacher-student 
interaction regarding academics, the use of appropriate "wait time" for 
student responses and frequent eye contact (Cohen, 1981, Ramey et al., 1982; 
Stallings, 1982). Finally, effective teachers project certain attitudes and 
expectations. These teachers are cooperative, hold high expectations for 
themselves as well as their students, and provide positive reinforcement to 
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others (Armor et al., 1976; Berliner, 1979; Brookover et al., 1979; Edmonds, 
1979; Rutter et al., 1979; Westbrook, 1982). 



Curriculum and Assessment Research . Research has pointed out the frequent 
mismatch between curriculum materials and the standardized tests used to 
assess student achievement (Crock & Scott, 1982; Carlberg, 1980; Floden et 
al., 1978). The absence of a relationship between textbooks and standardized 
tests is an important concern when selecting texts and tests, interpreting 
test results, and when evaluating instructional programs. Effective 
curriculum is "aligned" through the matching of classroom instruction with 
instructional intents and assessment of student learning (Brock and Schott, 
1982). 

Finally, the extent to which the curriculum fosters the acquisition of 
higher level thinking skills is another variable relative to the 
effectiveness of classroom instruction. Research indicates that the academic 
tasks given to students dictate the level and type of learning for students 
(Doyle, 1983) and that most academic tasks presented to students do not 
foster high order thinking skills (Education Commiss ion "bf the States, 1982). 
The implication is that for effective teachers there must be more emphasis 
placed on direct instruction of higher order thinking skills (Beyer, 1984). 

Planned Educational Change Research . The literature on planned educational 
change is both voluminous and expanding at a seemingly exponential rate 
(Fullan, 1982; Glasen et al., 1983; Runkel and Harris, 1983; Van Meter, 
1984). Consequently, no attempt to summarize it will be made here. There 
are some aspects of that literature, however, which are particularly 
pertinent to a systematic school improvement effort aimed at building and 
classroom level practitioners. These include descriptions of the generic 
stages of a planned change effort, the appropriate setting and effective 
means for stimulating planned change in schools, the nature and degrees of 
involvement of the various participants in the effort, and the basic 
character of planned change as a long-term endeavor. 

The stages of planned change include: orientation or recognition of 
needs; the initial implementation of the change to fit the context; and 
evaluation of the change effort (Berman and McLaughlin, 1974, 1976, 1978; 
Berman and Pauley, 1975; Crandell et al., 1982; Farrar and Cohen, 1980; 
Fullan and Pomfret, 1977; Louis et al., 1981). Researchers have reported 
that the school is the logical unit for planned educational change (Brookover 
et al., 1979; Brookover and Lezotte, 1979; Hichman et al., 1983; Rutter et 
al., 1979). Further, a successful change effort requires a long-term 
•:omraitment, which includes renewal activities (Hail and Loucks, 1977; Joyce 
and Showers , 1982) . 

The literature also describes the importance of participation by key 
educators in the change effort. The principal's leadership and participation 
are of critical importance to the success of planned change (Austin, 1981; 
Berman and McLaughlin, 1976; Corbett, 1982; Brake and Miller, 1982; Hager and 
Scarr, 1983; Kelly, 1960; Keys and Bartunek, 1979; Little, 1981; Nickolson 
and Tracy, 1982; Porter, 1980; Rosenblum and Jastrzab, 1980; Snycler, 1963; 
Williams, 1980). Further, several studiei also describe the role of the 
building principal in relation to effective schools, student achievement and 
school climate (Berman and McLaughlin, 1^76; Brookover and Lezotte, 1979; 
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Ellett and Walberg, 1979; Hager and Scarr, 1983; Liethwood and Montgomery, 
1982; Schneider, 1981; Snyder, 1983; Young, 1980). 



The effect of superintendent and central office staff involvement in the 
planned change processes has also been reported (Keys and Bartunek, 1979; 
Runkel and Schmuck, 1974, 1976; Schmuck et al . , 1975); their participation 
appears crucial to the success of any innovation (Heckxnan, Oakes and 
Sirotnik, 1983; Milstein, 1978). Finally, the literature suggests that 
participant planning and decision-making should accompany any change effort 
(Berman and Pauley, 1975; Klausmeier, 1982; Lawrence et al . , 1974; 
Nauraann-E^tienne, 1974; Parish and Arends, 1983; Parker, 1980). 

Research on Staff Development . Research shows achievement gains are 
associated with ongoing, systematic inservice training programs for teachers 
(Armor et al., 1976; Joyce and Showers, 1981; Trisman et al., 1976). Studies 
have shown that school-based staff development programs have a greater 
influence on teacher behaviors and attitudes than non-school-based programs. 
Similarly, inservice trainings are most effective when conducted at the 
individual school site (Henderson and Perry, 1981; Hersh- ^t al., 1981; Joyce, 
1981; House, 1974; Porter, 1980). It hi'S also been found that inservice 
education can be an effective change intervention for bringing about school 
improvement (Berman and Pauley, 1975; Fullan and Pomfret, 1977; Harris, 
1980). Research indicates that the content for an effective inservice should 
include an interpersonal relationship component as well as task or content 
components (Runkel and Schmuck, 1976; Schmuck et al . , 1975). Effective staff 
development programs also include theory, demonstration, practice and 
coaching activities so that the implementation process is supported (Bird and 
Little, 1983; Joyce, 1981; Joyce and Showers, 1982). 

The McREL ESP program attempts to organize all the major categories of 
the research cited above in a conceptual framework which guides the Effective 
Schools Program. That framework organizes the literature into three distinct 
areas: 1) Teaching and Instruction, 2) Building Level Leadership and 3) 
Organization, and Curriculum/Assessment Relationship. This framework is 
graphically portrayed below in Figure 1. 



Figure 1 




PROGRAM GOALS 



The intent of McREL's Effective Schools Program is best described 
through the ESP goals; to provide successful learning opportunities to al 1 
students who attended school; and to develop and/or enhance a school 
improvement process to foster the development of cooperative, self-renewing 
improvement efforts guided by site-based leadership teams. 

To accomplish these goals, the ESP has five basic objectives: 

o help participants gain knowledge about the effective schools and 
teaching research 

o help part icipant s master diagnostic procedures to compare their 
school with the characteristics of an effective school, as 
described in the research 

o help participants select alternative strategies to improve 
performance in areas of need ind icat ed by the analysis of 
d iagnost ic procedures 

o help part ic ipant s implement the relevant strategies 

o help participants develop an assessment system to document 
improvement s in student achievement and other performance 
outcomes resulting from ESP activities 

Meeting these objectives requires commitmenLj from both the McREL and the 
school staffs. Through the ESP, McREL provides teachers and administrators 
with an orientation to t research on instruct ional ly effective schools, and 
translates this research into clearly defined action steps so that 
improvements can be accomplished within the individual school settings. An 
initial step in the school improvement process is the development of the 
building level leadership teams. Each leadership team represents a school 
building within the district and is composed of the principal and four to 
eight teachers from that building. Teachers typically are drawn from the 
building's various grade, content, and specialist areas. Through the 
McREL-ESP, team members become leaders in fostering school collegial ity , 
carrying out the tasks of identifying school goals and priorities, designing 
a leadership team, selecting activities which lead to increased 
effectiveness, and assessing their own progress in implementing changes. 

PROGRAM DELIVERY 

The McREL-ESP is delivered through four one-day workshops for building 
level teams. The workshop sessions are scheduled a month or more apart and 
usually occur in September, October, November, and February. Four half-day 
meetings are scheduled for administrator development; the meetings are 
conducted at intervals between September and May. Two days are scheduled for 
follow-up site visits at the individual school buildings. Site visits are 
scheduled for the Spring or Fall following the workshop sessions at the 
request of the building teams. During site visits McREL staff members 
provide additional support materials, act as coaches for team me;nbers , and 
assist teams in implementing their leadership plans for school improvement. 

-6- 
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The content of the leadership team training sessions integrates the 
three areas of research within the conceptual model portrayed above. Five 
major topics are addressed in the ESP training sessions commonly referred to 
as "workshops:** 

o Teaching and Instruction 

o Building Level Leadership and Organization 
o Cur ricul urn/ Assessment Relat ionship 
o Facilitation and Change 
o Planning 

These topics are covered through a series of 25 program activities. While 
there is some variation from site to site, the schedule of activities listed 
below is generally followed: 

Session I 

Activity 1: 

Activity 2: 
Activity 3: 

Activity 4: 
Act ivity 5: 
Ac t iv i t y 6 : 
Activity 7: 
Ac t iv i t y 8 : 

Session II 

Activity 9: Debriefing the Results of the School Improvement 

Quest ionnaire 
Act ivity 10: Effective Inst rue t ion 

Activity 11: Classroom Organization and Management and Beginning 
the School Year 

Activity 12: Coaching and Communication: Building Support Systems 
Activity 13: Action Planning: Preparing for Change 

Session III 

Activity 14: Debriefing Time Data 

Activity 15: Developing the Academic Efficiency Index 

Activity 16: Discipline: Reducing Disruptions 

Activity 17: Expectations: The Subtle Difference 

Ac t iv i t y 18: Mo t i v a t i on 

Activity 19: Curriculum Alignment Diagnosis, Assessment, and 

Instruct ional Management 

Activity 20: Planning: The Next Step 

Session IV 

Activity 21: Ov'erview: Higher Order Thinking Skills Development 

Activity 22: Summary of t*r»e Pieces: Putting It All Together 

Activity 23: Team Building and Cooperation 
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Introduction, Data Collection, Materials 
Disseminat ion 

Introduction to Research and Development 

Overview of the Literature on School and Classroom 

Effect iveness 

Orientation of the Effective Schools Program 
Time in Schools 

Building Level Leadership and Organization 
Introduct ion to Curriculum/ Assessment Relar ionship 
The Change Process in Schools: Your Leadership 



Activity 2A: Developing the Leadership Plan 
Activity 25: Wrap-up: This Is Just the Beginning 



Below is a brief narrative description of the coverage of the five topics. 

Teaching and Instruct ion . A key concept within the teaching and 
instruction topic is the effect of teacher expectations on student 
performance. Other topics include task analysis, instructional strategies, 
instructional models, classroom management, student management, 
reinforcement, motivation, and diagnosis of student needs. 

Building Level Leadership and Organization . This area draws on the 
research that describes characteristics associated with school effectiveness 
at the building level, e.g., building leadership, school climate, school 
organization, building goals, collegiality and parent involvement. 

Administrator development is also a unit within this topic area. The 
administrator-training sessions focus on the roles of the building principal. 
Building administrators assess their own behavior in an 'attempt to identify 
personal characteristics consistent with those of effective building leaders. 
The McREL-ESP provides techniques through which administrators determine how 
they spend their professional time, and how building staff members perceive 
the principal's actions. 

Curriculum/Assessment Rel at ionship . Key concepts in this area include 
test content analysis, test result analysis, curriculum alignment, textbook 
content analysis, test-taking skills and higher order thinking skills. The 
Content Determination Research Project and the Curriculum Al ignment Projec . 
are shared as alternative ways of looking at curriculum and assessment. Team 
members examine tests to determine the content areas measured and then align 
their instruction with those areas. Participants also acquire skills to 
determine which content areas should be taught, the materials needed to teach 
each content area, the alignment of content areas and various measures used 
to evaluate the level of student achievement. 

Facilitation and Change . Within this topic area, team members gain an 
understanding of their roles as leaders in the planned change effort in their 
schools. They learn about barriers to school change, effective listening 
techniques, conditions for creating change, cooperation, and collegiality. 
By becoming "coaches" for other staff members, team members model behaviors 
necessary for successful change. The McREL-ESP objective relative to this 
area is that team members, as leaders, realize that effective communicat i :^n 
and collaborative participation in change activities are essential to school 
improvements. Tne leadership team guides implementation with that 
realization in mine. 

Planning . Following data collection efforts, building teams develop a 
leadership plan to address the team's leadership role during the improvement 
process. The team members describe the present state of their efforts and 
the team ^nd school goals to be accomplished in one year. To reach these 
goals, the team aeiermines manageable improvement steps, and includes these 
in the plan. The team plans are systematically designed to provide a base 
for leadership decisions regarding the projected changes. The key issue here 
is that the team vriies a plan for its own work, not a plan for the entire 
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school faculty. If the learn determines that a building-wide, school 
improvement plan is important, team members design activities to include the 
entire faculty in the development of that plan. 

PROGRAM ACTIVITIES 

Throughout the ESP McREL staff has a role similar to that which Crandall 
(1982) calls the "total change agent." The total change agent maintains a 
long-term, comprehensive relationship with the local educational agency as it 
passes through the phases of growth. The role of the agent varies from time 
to time, and is responsive to the sometimes idiosyncratic nature of various 
schools and districts. The total change agent is also responsive to the 
various stages of change within a school or district. During the ESP, McREL 
staff monitor a number of distinct phases: 

1) Orientat ion . During initial communications, McREL staff members meet 
with key rj*»c is ion-makers in the district. At this time, a variety of 
strategie.- are used to share information such as presentations, meetings, and 
written documents. In most cases, a McREL staff member presents an overview 
of the research on inst ruct ional ly effective schools, and provides 
suggestions regarding possible future activities in which district personnel 
might engage. The key issues during this orientation phase include 
establishing local commitment and ownership, and providing a professional 
perspective or atmosphere in which educators with varying kinds of knowledge 
and expertise can join together to work toward a common goal. This 
perspective differs from a strictly consultive arrangement because each 
•'player" — McREL staff member, central office administrator, board member, 
building level educator — is perceived to bring expertise and \nformation to 
the effort. This sets a framework for the next phase. 

2) Program Planning . After the orientation phase, McREL personnel and 
school district representatives engage in joint planning. The meeting 
schedule, participant selection, material development, and assessment 
activities are examples of the topics addressed by the planning team. The 
key issue in joint planning is the meldirg of the McREL-ESP requirements with 
the individual needs of each district. Tnus , negotiation becomes a natural 
part of planning discussions. For example, the McREL program specifies that 
leadership teams represent individual school buildings and that they must 
include the principal as a team member. Nevertheless, some districts iright 
have two buildings run by only one principal. The issue is discussed by the 
planning team and a decision is jointly reached . 

It is important to note that there are some McREL-ESP requirements which 
are not negotiable due to their important . These requirements include: 

o Team size must be fewer than nine - eight teachers and the 
principal. Leadership teams and large committees are not the 
same. Too large a group cannot develop the unity needed to 
carry out leadership functions . 

o Central office support is essential. A single school building 
staff wanting to participate without the support of its central 
administration has little chance of carrying out the chances 
wh ich are necessary to produce resul t s . Quest ions of 
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curriculum^ testing^ grading and evaluation are examples of 
critically important program topics which are not specific to 
individual buildings . 

o Leadership development training days must be spaced four to six 
weeks apart. Participants use time belween sessions to reflect » 
impl ement suggested act iv it ies » assess needs and observe 
instruction. On the other hand, too much time (for example, a 
summer) interferes with the momentum of the team's development, 

o A maximum of fifty participants attend each leadership team 
training session. The interactive nature of activites and the 
need for group discussion demand the total group remain a 
manageable size, 

o Leadership team members must administer the assessment 

instruments designed by the Laboratory as part of the Effective 
School Program. Assessment is an integral part of all aspects 
of the program and cannot be avoided, 

o The leadership team membership must remain constant. Team 
members must attend all the sessions unless an emergency 
interferes, in which case, no substitutions are made. The unity 
of the team is again the issue, 

3) Leadership Development . Unlike a "trainer of trainers" approach to 
school improvement, the core of the McREL-ESP approach is the development of 
building leadership teams. Leadership development differs from "trainer of 
trainers" in that the participants are never expected to reproduce or 
duplicate the training in which they have participated. Instead, the team 
develops or enhances its leadership skills so it can guide site-specific 
efforts to implement the research on instruct ionally effective schools and 
classrooms. The McREL-ESP approach to this leadership includes: 

Information Presentations , These activities are used to establish 
a common understanding of the literature base for effective 
school ing . Common knowledge about the content is essential for any 
dec is ion-making team, 

Demonst rat ions . The McREL staff member models various leadership 
behaviors for the teams. Problem-solving, decision-making, 
supervising , "coaching" and team building are included . 
Additionally, the McREL presenter demonstrates the use of 
instructional methods, observation techniques, and other strategies 
for implementing strategies. 

Feedback , Just as teachers need feedback while trying to enhance 
their instructional skills, the leadership team members need 
feedback as they carry on their work. The McREL-ESP staff members 
act as a sounding board, by planning with, talking with, and giving 
feedback to leadership team members. 

Planning . Planning is a key component of the ESP program. It is 
important to note that leadership teams do not develop a 
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EVALUATION DESICN 

As dctcribtd in the Interim Report, the evaluation of the McREL 
Effective Schools Program is based on a linear model of differentiated 
effects to various groups within a school or district. That model is 
presented in Figure 2. 

Figure 2 
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percept ions 
and skills 



Level 


B 


w — 

Charges in 

behavior 

onsite 




Level 


C 


— t 


Changes in the 
^ building climate, 

policies and practices 


Level 


D 




^^^^ Changes in the 
^^i^ Classrooms 

Management and Instruct ion 



Level E 



Changes in 
student mo 
t ivat ion , 
engagement , 
success 
rate.s. etc. 



Level F 



Changes in 
student 
achievement 
and Adademic 
self-confi- 
dence 
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As indicated in Figure 2, the ESP is hypothesized to have a layered 
effect. The stages in Figure 2 represent the different groups affected by 
the ESP. The levels represent the different types of change within those 
various groups. 

The evaluation of the ESP was designed around the stages and levels 
depicted in Figure 2. More specifically, the ESP evaluation can be 
conceptualized as encompassing seven different components. 

Stage I: Level A: The design of the training workshop 



Stage II: Level A: 



Changes in knowledge, awareness, perception and 
skills of participants 



Stage 


II: 


Level 


B: 


Changes 


in 


Stage 


III: 


Level 


C: 


Changes 


in 


Stage 


IV: 


Level 


D: 


Changes 


in 


Stage 


V: 


Level 


E: 


Changes 


in 



etc . 



Stage V: Level F: Changes in student achievement and academic self- 
confidence 



Section III of this report describes the results for all components 
described above except for Stage I, Level A. The evaluation for that 
component would necessarily focus on the extent to which the McREL Effective 
Schools Program incorporated current research and theory on effective 
schooling. Section I of this report contains a brief review of that research 
and theory base. However, that review should be considered only as an 
executive summary of the research and theory base of the McREL Effective 
Schools Program. The four years prior to the 1984-85 academic year saw 
extensive efforts to compile and organize the effective school literature and 
involved many nationally known leaders in the effective school movement. 
Consequently, for the purposes of this report the assumption is being made 
that Stage I: Level A has received a thorough evaluation in previous years. 

Figure 3 contains a summary of the stages and levels involved in the 
1984-85 evaluat ion along with the evaluation questions and data collection 
techniques . 



-13- 



Figure 3 



STAGE 


LEVEL 


EVALUATION QUESTIONS 


DATA COLLECTION 


II 

ESP 
PARTICIPANT 


A 

Changes in ESF 
participant* s 
knowledge , 
awareness » and 
percept ions 


Do participants demon- 
demonstrate an increase 
in knowledge? 

Do participants feel the 
training was beneficial? 


Cognitive Pre/ 
Post Tests 

Workshop 
Evaluation Form 




B 

Changes in ESP 
part ic ipant * s 
behavior on 
site 


Do program participants 
use the skills and 
strategies after the 
training? 


Leadership Plans 
Interviews 


III 
BUILDING 
VARIABLES 


C 

Changes in 

building 

c limate , pol- 

ic ies and pr ac- 

t ices 


To what extent are the 
staff members from the 
schools involved in the 
ESP engaged in a school- 
wide improvement effort ? 


Site Visit 
Interviews 


IV 

CLASSROOM 
VARIABLES 


D 

Changes within 
c lassrooms 


Do staff nidmbers utilize 
the school improvement 
techniques designed to 
affect classroom 
variables 


Site Visits 
Interviews 


V 

STUDENT 
VARIABLES 


E 

Changes in 
student 
mot ivat ion » 
engagement and 
success 


Have the appl ic at ions of 
ESP techniques caused 
improvement s in student 
mot ivat ion , engagement 
and success? 


Interviews 




F 

Changes in 

student 

achievement 


Has student achievement 
increased? 

Has inequity of 

effectiveness 

decreased? 


Informal 



DATA COLLECTION 

Although the data collection techniques mentioned in Figure 3 are 
described in depth in their respective subsections of Section III of this 
report, below is a brief description of each technique: 

COGNITIVE TEST OF PARTICIPANT KNOWLEDGE 

McREL uses pre and post cognitive tests to assess participant knowledge 
of the research and theory presented in the ESP. The specific knowledge 
measured in the test focuses directly on information available in the McREL 
Quality Schools Folios (See Appendix A). 
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McREL ESP WORKSHOP EVALUATION FORM 

To assist McREL staff with efforts to refine and improve the ESP 
training program, workshop evaluation forms are completed by each 
participant. This form seeks participant opinions concerning workshop 
quality, significance and content. The form also asks their opinions of the 
quality of the presentations, materials, opportunities for feedback and 
various logistical issues. The completed forms are received at the 
conclusion of each of the four training sessions (See Appendix B) . 

LEADERSHIP PLANS 

At the conclusion of the training sessions, team members are requested 
to develop a one-year program for implementing the effective schools research 
within their school district, McREL staff meet with the planning team to set 
objectives, schedules and activities. The teams are asked to submit the 
completed plan to McREL for feedback and analysis; however, this is 
voluntary. 

SITE VISITS 

To assist school teams with plan implementation, McREL staff provide two 
follow-up site visits. During these visits, observat ions- are conducted to 
determine the extent to which changes relative to the effective schools 
variables have occurred, and assistance is provided through coaching, 
modeling and dissemination of relevant materials. Conferences are held with 
the school teams following the visit to provit'e feedback and to assist with 
team concerns, 

INTERVIEWS 

To determine the intermediate and long-term effects of the ESP training, 
a follow-up phone interview was conducted with previous participants. The 
respondents were asked specific questions concerning changes in behaviors, 
practices and policies and if these changes are a result of the ESP training. 
The interview also include questions regarding changes which have occurred 
within classrooms, and the impact this has had on students and student 
achievement. The respondents were also asked to rate the ESP's overall 
ef f ec t iveness , 

ACADEMIC EFFICIENCY INDEX (AEI) 

The AEI instrument enables participants to develop an estimate of the 
efficiency of instructional time. Two sub-instruments, the Classroom 
Observation Worksheet and the Academic Efficiency Worksheet are used as 
recording devices. Observers and team members record the amount of time 
available and the use of that time during the school day. The AEI provides 
formulas to estimate the percentage of the school day that student© are 
engaged in academic activities, and can be utilized repeatedly to determine 
longitudinal increases in engagement rates (See Appendix C) . 

DESCRIBING YOUR SCHOOL'S CHARACTERISTICS (DYSC) 

The DYSC is a questionnaire designed to measure teachers' and 
principals' perceptions of the characteristics that describe their school. 
Statements associated with the effective schools research are included in the 
questionnaire. Respondents then indicate whether they perceive each statemen 
to be true for their school. The statements are followed by a five point 
response scale ranging from always true to never true. The 63 items are 
grouped into seven clusters or categories relative to the effective schools 
research. The items are computer scored snd average scores are obtained for 
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each cluster. The average cluster scores from all the teacher questionnaires 
are compared to the average cluster scores from all the administrator 
questionnaires. The final results are returned to the school along with an 
explanation of their meaning (See Appendix D) . 

SUBJECTS 

There were ten sites used for the 1984-85 ESP evaluation. They are 
listed below along with approximate training days. 

ESP TRAINING SESSIONS 





1984 


-1985 






Anakenny, lA 


9/84 


10/84 


1/85 


3/84 


Cedar Rapids, lA 


8/84 


10/84 


2/85 


4/85 


Cylinder, lA 


9/84 


10/84 


3/85 


4/85 


Detroit, MI 


3/85 


4/85 


5/85 


8/85 


Emporia, KS 


10/85 


11/84 


1/85 


3/85 


Ft. Madison, LA 


11/84 


1/85 


2/85 


3/85 


Liberty, MO 


10/84 


12/84 


1/85 


3/85 


Nebraska ESU, N£ 


9/84 


10/84 


2/85 


4/85 


Sioux Falls, lA 


10/84 


2/85 


3/85 


5/85 


Winfield, KS 


10/84 


11/84 


1/85 


3/85 


These districts 


represent urban. 


suburban , 


and rural 


populations. Each 



training is usually conducted with teams of 35 teachers and administrators; 
however, in some cases trainings are conducted with as many as 50 
participants. Hence the 1984-85 evaluation was based on data from ten 
districts which encompassed over 350 teachers. 

LIMITATIONS 

As alluded to in the introduction to this report, there were many 
constraints on and limitations to the ESP evaluation. 

First, like most complex change programs, the ESP has diffuse goals and 
objectives. The cumulative effect of the series and interactions that 
represent complex training events is dependent upon the change in variables 
which the program affects indirectly. Due to the complexity of the change 
environment, the program is only one of a myriad of factors influencing the 
knowledge, perceptions, beliefs and behaviors of participants. Therefore, 
the direct affects of the ESP is only one factor influencing the ultimate 
change in classrooms and buildings. Because of this complexity, we did not 
nor could not experimentally isolate the intended effects of the ESP. 
Rather, we recognize that positive changes and interesting patterns of effect 
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may be "contaminated*' and/or diluted by non-ESP occurrences. Additionally, 
some effects may not be captured by our assessment instruments. Our solution 
is to examine the weight of available evidence across sites and measures. 
That is, we seek to show that despite the variability of contexts, certain 
effects and changes appear most or all of the time in ESP training sites, 

A second constraint on evaluating the ESP is McREL's desire to refrain 
from placing too great a response burden on participants. Assessment 
requirements of the program itself, probably approach participants' upper 
limits of tolerance. They are "paying customers" in most cases, and are not 
always amenable to providing data not needed for the training. For this 
reason, the 1984-85 evaluation relies on the data produced through training 
rather than on additional data gathered solely for evaluation purposed. In 
addition, we use an unbalanced matrix, pre-exper imental sampling design; each 
of the data sources used is available from only a subsaraple of sites, 
selected in part on the basis of their willingness to provide the information 
we need. As this data is provided voluntarily, it is also important to 
assure anonymity whenever possible. Again, we rely on weight of evidence, or 
triangulation, rather than a pure experimental effect, to inform us about 
what the ESP is accomplishing. 

An additional feature of the program reinforces the choice of a ^ 
non-probability sampling approach; that is, the treatment or intervention is 
fluid, and in some respects, changeable. Trainers intentionally adapt their 
efforts to the idiosyncrasies of sites and participant groups, rather than 
treating all sites the same. Within bounds, the trainers differ in the 
approaches they use, emphasizing those they do best. The program itself is 
tailored to the needs and goals at each site; essentially, each site receives 
a custom program. And, the ESP is constantly evolving to accommodate and/or 
capitalize on new research findings, newly-discovered training approaches, 
new areas of interest. Therefore, the available instruments techniques are 
used according to their appropr ic.teness at any given site. 
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SECTION 111 
RESULTS 



In this section are presented the results from the six stages and levels 
described in Section II as components of the 1984-85 ESP evaluation. Each 
stage and level is discussed separately. 

STAGE II: Level A 

The evaluation questions for Stage II: Level A are: 

Do participants demonstrate an increase in knowledge as measured by 
a cognitive test? 

Do participants feel the training is beneficial? 

These are discussed separately below: 

Do Participants Demonstrate an Increase in Knowledge ? 

Data for this evaluation question was gathered using, a ''cognitive test" 
or questio.maire. The questionnaire was developed to assess the amount of 
knowledge related to classroom research that participants had before and 
after the training series. The specific knowledge included in the 
questionaire focuses directly on information available in the McREL Quality 
Schools Folios on Time, Instruction, Discipline, Beginning the School Year, 
Motivation, and Expectations. Since the ESP training is much broader and 
comprehensive in scope than are the Folios, the questionnaire is perhaps best 
understood as a measure of comprehension of the Folio contents. 

Every ESP participant trained during the 1984-85 school year was given a 
cognitive test covering information presented in the folios. The 
distribution of respondents having both pre and post scores is presented 
below: 

Distribution of Respondents Having Both 
Pre- and Post-training Tests by Training Site 

Site Number Respondents 

Emporia 24 

Fort Madison 23 

Liberty 32 

Winfield 9 

The tests were given during the first and final workshop sessions, and 
were coded with the participants' names or self-selected codes so each 
participant's pre-test and post-test could be matched. The answers to the 
Pre-Workshop and Post-Workshop Questionnaires were scored as incorrect or 
correct using a key developed by the ESP trainers. 
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The results for the pre/post comparison using fl two-tailed, t-test for 
dependent groups are presented below: 



PRETEST 




(N-88) 






HIGHEST SCORE 


21 


27 (points possible=38) 


LOWEST SCORE 


0 


5 


RANGE 


21 


22 


MEAN 


9.13 


16.83 


STANDARD DEVIATION 


A. 35 


6.26 


STANDARD ERROR 


.A6 


.67 



DEPENDENT t-TEST 



(one-tailed) 






MEAN DIFFERENCE 


7.67 


degrees of freedom=87 


STANDARD DEVIATION 
OF DIFFERENCE 


6. 11 


d =.01 
critical value=2.39 


DIFFERENCE 


.65 




t-STATISTIC 


11.8 





As indicated above, a highly significant difference between the pre-test 
and post-test scores was found. The critical t-statistic was significant at 
the 99% confidence level providing strong evidence of a substantial increase 
in participants' knowledge. This demonstrates an increase in knowledge on 
the part of participants from the first to the final ESP workshops. In 
effect then, the answer to the first evaluation question appears to be 
"Yes-participants do demonstrate an increase in knowledge," 

As indicated above, these results are based on responses from four 
districts only. Difficulties in obtaining results include: two districts 
could not be scored because participants did not identify themselves on pre 
and post tests; three districts changed (corrected) answers following trainer 
feedback and were, therefore, invalid. 

Following are some suggestions for continued use of the cognitive test. 

1) Complete an item analysis on the new test items. Select the items 
which best discriminate between the most knowledgeable and least 
knowledgeable. Attempt to keep internal consistency between ,80 and 
.90. 
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2) Participants should be provided with corrective feedback about their 
responses as quickly as possible after the post-test. Again this 
models behavior which the ESP is designed to encourage. However, to 
obtain comparative scores, participants should be encouraged to 
identify their tests and to keep their nnswers intact. 

Do Participants Feel the Training Is Beneficial ? 

For the above evaluation question. Workshop Evaluation Forms wure used 
(Appendix B) . More specifically, following each training session, 
participants were asked to complete an evaluation form designed to assess the 
workshop's strengths and weaknesses. Participants were asked to rate 
(5»Excellent ; l«Poor) the effectiveness of the following workshop 
components: 

-Act ivi ties /Present at ions 
-Materials 

-Opportunities for Participation 
-Opportunit ies for Feedback 
-Organization 

-Usefulness of Information 

-Relevancy of Information for the Organization 
-Usefulness of the Workshop, Personally 

In addition, participants were asked to suggest improvements and to comment 
on strengths and weaknesses of the workshop. 

Participant Ratings of Workshop Sessions 

The training sessions were rateo very highly by participants. As indicated 
in Figure 4 below, the average rating across all districts and sessions was 
4.20. Only 7 out of 24 sessions were rated below an average of 4.00. The 
variation across districts and sessions was small with the lowest average 
rating for any session being 3.83, the highest being 4.69. 

Figures A through E in Appendix E show the average ratings for each aspect of 
the workshops, by district and session. For example, for the Detroit Public 
Schools (Figure A), participants gave high ratings to all aspects of each 
workshop session. Participancs felt that the activities, materials, and 
organization vere very good (4.47, 4.73, 4.43); that the opportunities for 
participation and feedback were quite good (4.47, 4.46); that the information 
was extremely useful and relevant (4.87, 4.68); and that in general the 
workshop was quite useful to them personally (4.58). Figures B through E 
show similar results for the other five districts. 
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'84-'85 ESP Training Evaluations x District and Session 

N " 5,633* 





Session 


Session 


Session 


Session 






1 


11 


III 


IV 


TOTAL 


Detroit 


4.69 


A. 38 


4.54 


4.69 


4.58 


Emporia 


4.13 


3.83 


3.99 


3.88 


3.96 


Ft, Madison 


4.37 


A. 46 


4.51 


4.64 


4.49 


Liberty 


li 24 


3.94 


3.99 


4.11 


4.07 


Sioux Falls 


3.98 


3.73 


4.35 


4.49 


4.17 


Winf ield 


4.18 


A. OP 


4.00 


4.17 


4.08 


Total 


4.28 


A.o; 


4.18 


4.27 


4.20 


^Excellent « 5 


: Poor = 1 
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Perceived Strengths and Weaknesses 
of the ESP Training Sessions 



The workshop evaluation form asked participants to suggest improvements in 
workshop design and to discuss the workshops' strengt.ls and weaknesses. 
Figures A through F in Appendix F provide district summaries by training 
session of participants' suggestions for improvement and their perceptions of 
the workshop's strengths and limitations* A discussion of these suggestions 
and comments follows* 

1 . Perceived Strengths of the Training Sessions 

Across all districts and sessions, participants identified the richness 
and extensiveness of the information provided as the major strength of 
the workshops. General examples mentioned frequently included models, 
strategies, and research findings. More specific examples included the 
information on discipline, motivation, instruction, time management and 
assessment. Student Team Learning, expectations, and coaching. Other 
frequently listed strengths included: ?~ 



2. Perceived Weaknesses and Suggestions for Improvement 

Across all districts and sessions, the major weakness perceived by 
participants was related to time limitations. Participants felt that 
there simply wasn't enough time to absorb the extensive amount of 
information in the short time provided. They also felt a need for more 
discussion, sharing, and interaction. Other perceived weaknesses 
included problems with the organization of some of the materials and 
handouts, quality of some of the visuals, and a need for more directions 
and "how to" suggestions. Some participants felt a need for more clarity 
around program purposes, outcomes, and expectations. 

3. Other Participant Comments 

Figures A through F in Appendix G list other participant comments about 
the workshop sessions. The comments are both general and specific, and 
cover a wide range of topics, some of which address district-specific 
issues that are not under the control of ESP trainers. As a rough 
measure of the direction of these data, each comment was assigned one of 
the f ol lowing ratings : 

+ = positive, re. ESP 

0 = neutral/not applicable 

- = negative, re. ESP 

Figure 5 summarizes these data across districts and sessions. Overall, 
there were 75 positive comments suggesting the usefulness of, or 
appreciation for, the workshops; 32 neutral or non-applicable comments; 
and 44 comments suggesting changes/ improvements in workshop design or 
implementation. 



-quality and positive attitude of presenters 
-opportunities for sharing and interaction 
-reaffirmation of ideas and practices already in place 
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Figure 5 

Positive, Neutral, and Negative Nature 
of Other Comments 



Session I 



Session II 



Session III Session IV TOTAL 



Detroit 

positive 

neutral 

negative 

Emporia 

positive 
neutral 
negat ive 

Ft. Madison 
positive 
neutral 
negat ive 

Liberty 

positive 
neutral 
negat ive 

Sioux Falls 
positive 
neutral 
negat ive 

Winf ield 

positive 
neutral 
negat ive 



2 
0 
1 



6 
4 
5 



5 
2 
1 



6 
7 
6 



1 
1 
0 



8 
3 
1 



1 
1 
0 



1 
3 
3 



4 
0 
0 



2 

0 
4 



0 

0 
2 



0 
0 
2 



1 
0 
2 



3 
1 
0 



3 
0 
0 



3 
0 
4 



1 
0 
0 



1 
0 
0 



2 
3 
1 



4 
2 
3 



5 
1 
3 



3 
0 
5 



4 
0 
1 



4 
0 
1 



6 
4 
4 



14 
10 

11 



17 
3 
4 



14 
7 
19 



6 

1 
3 



6 

1 
3 



Total 



positive 
neutral 
negat ive 



28 
17 
14 



9 
5 

11 



15 
1 
6 



23 
9 

13 



75 
32 
44 



In summary, participants expressed very positive perceptions of the ESP 
training workshops. The following aspects of the sessions were rated very 
highly: 

-Activities, materials, and organization 
-Opportunities for participation and feedback 
-Usefulness and relevancy of the information 

The major perceived strengths of the workshops were the quality of 
information and materials provided, the presenters, the opportunities for 
sharing and interaction, and the reaffirmation of ideas, and practices already 
in place. The major weakness was the time limitation, in light of all of the 
information to cover, particularly as the limits impinged on opportunities 
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for discussion and interaction. Other participant comments provided very 
positive perceptions of the program and its activities, useful suggestions 
for future workshops, and great enthusiasm for the program's potential for 
improving schooling at the building and district levels. 

STAGE II: Level B 

The effective schools research on the related materials provided during 
workshop sessions enables participants to increase their knowledge of 
effective school characteristics. However, the acquired knowledge must be 
applied toward the existing school environment so that further behavior 
change can occur. Such change occurs at Stage II: Level B. The evaluation 
question for this stage and level is: 

-Do program participants use the skills and strategies after the 
training? 

The long-term development plans, called "Leadership Plans," served as the 
data to answer this evaluation question. Leadership Plan development 
demonstrates participants' ability to state and implement building-level 
improvement strategies. Following training the McREL trainer meets with the 
leadership teams to assist with goal and objective development. As a result 
of this meeting a leadership plan is developed by each team for the purpose 
of operat ionalizing and sharing the information they gained during the 
workshops with the rest of the building staff. Eight Leadership Plans 
1984-85 sites were available. These plans were evaluated to measure the 
initial changes in on-site behavior. 

Each plan was reviewed by three McREL staff members and was analyzed for 
inclusion of the characteristics listed below. For each characteristic the 
following rating scale was used: 0 = no evidence of characteristic; 5 = 
clear evidence of characteristic. 

1. Is the purpose of the plan clearly stated or implied; i.e., the 
purpose addresses the teams' leadership role by describing 
manageable steps to improvement which they can facilitate in their 
school? 

2. Are the goals (or where they want to be as a team in Spring 1986) 
clearly stated? 

3. Are the goals consistent with the information presented during the 
ESP workshops? This information includes assessment data collected 
during the ESP as well as the discussion of the effective schools 
research . 

4. Are the activities specifically stated and related to the goals? 

5. Are the criteria for determining when the goals have been reached 
speci f ical ly st ated? 

6. Could the plan be used by a McREL trainer during a 1985-86 site 
visit to provide feedback about progress to the team? 
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•vtrift ch«r«cttriit ici icorct «nd tht avtragt ovtrall scores %frrc 
ccUulatH (or ttch plM «nd tht tntirt stt of plans. Tht rtsulis of thest 
•Atlytts art rtporttd btlow. 

CHARACTER 1ST ICS 



Total Mean 







1 


2 


3 


4 


5 


6 


Ptr Plan 




A 


3.33 


3.00 


3.33 


1.33 


6.60 




2.Q5 




r" 


4.00 


-TIT 




3.66 




<».uu 


3.66 


f 


c 


S.M 




* .vw 


2.66 


1.60 


2.66 


1.8S 


L 


5 




4.66 




^ • w 




5.00 


— nr 


A 


■ 
• 


0.00 


0.33 




1.00 


0.00 




-OF 


N 






3.00 


J. J J 


4.00 




J ff vo 


i9 . w 


t 


6 


« .vO 


-JTJT 


9.66 




3.00 




W ff W7 




H 


2.66 


4.00 


4.00 


J .vu 


2.00 


i# ff vv 


i9 • A r 


Ovtrall 
















Ovtrall 


N#Ma 




3.04 


3.04 


3.25 


3.12 


1.92 


2.92 


2ff88 M««n for 

Entire Set 



TbeM rttultt indicate that the leadership plane received were of iB?xed 
quality. The total Man for Plan D equalled U.Si. Thie plan shoved 
sttbstantial evidence for each characteristic ff In contrast, the total isean 
for Flan C equalled 0.28. This plan provided little or no evidence of the 
target characteristics. Of the remaining plans » the average total scores for 
Flane A and C indicate low quality whereas the average total scores for Plans 
I9 C9 and H indicate higer quality. 

To further study the leadership plans the six target characteristics 
were rank ordered in terms of their frequency of inclusion in the leadership 
plane. This rank ordering is presented belowff 



Rank Order 



Overall Mean 





1 


Crittria for reaching goals 


1.92 




2 


Enablts trainer fetdback during 


2.92 






sitt visit 






3.5 


Cltarly stated purpose 


3.04 




3.5 


Goal coapleted date (clearly stated) 


3.04 




5 


Cltarly stated goal-related activities 


3.12 


High 


6 


ESP related goals 


3.25 



the plans are lower than expected due to the poor quality of Plan E. 
However I these scores provide information concerning the areas in vrhich 
schools may need planning assistance. These results do not include all ESP 
sites and consequently are probably not representative of all schools. It is 
suggested I however, that: 

1. Low ranking characteristics be emphasised in the development of 
future leadership plans; 

2. Leadership plans be more structured while still allowing for 
flexibility in content; 

3. Leadership plans be used to structure site visits which occur during 
the vear for %#hich the plans are written. This provision of 
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feedback should help achoolt decide which expectations were 
realistic and which were not. 

FOLLOW-UP INTERVIEWS 

In addition to the leadership plans, follow-up telephone interviews were 
conducted to evaluate Stage II: Level 3. These interviews were also used to 
evaluate other stages and levels (see Figure 3). 

Prior to conducting the interviews a stratified random sample was 
selected from a list of all schools completing the ESP training, not just 
those who participated in the training during the 1984-8i academic year. It 
was intended that both principals and teachers would be selected from this 
sample. However, due to the hectic schedule of classroom teachers, the 
interviews were conducted only with principals. Forty principals were 
contacted. The interview consisted of a series of questions designed to 
obtain data relative to specific stages and levels. One of those questions 
was specifically designed for Stage II: Level B: 

-Has your team developed a long-term building level -improvement plan? 

Of the 40 responding principals SOX indicated that their leadership team had 
developed and were actively implementing improvement plans. This was 
interpreted as an indication that the ESP effects are multi-year 
longitudinally. 

STAGE III: Level C 

The purpose of the Stage III: Level C evaluation was to determine the 
effects of the McREL ESP at the building level. The evaluation question for 
this stage and level was: 

-To what extent are the staff members for the schools involved in the 
ESP engaged in a school-wide improvement effort? 

Data for this evaluation question was gathered using site visits and 
fol low-up interviews . 

Site Visits are conducted after the four full-day Leadership Team 
training sessions. Since the ESP training sessions are scheduled over an 
entire academic year (Year One -- Development and Planning), 5?ite visits take 
place in Year Two (implementation) after team members have had an opportunity 
to plan and begin ESP implementation activities. The visits occur at the 
building team's request, which allows each team the flexibility to plan for 
change at a pace that is appropriate to their school. For some teams, site 
visits occur at the beginning of the school year as "back to school" 
activities are planned. Other teams prefer to wait until they have had a 
chance to start the new school year. They then invite in the McREL staff. 

Each site visit is unique to the particular needs of the host school. 
During the site visit, the relationship of the Laboratory staff to the 
Leadership Team is supportive. The McREL visitor provides feedback to the 
team by answering questions concerning implementation strategies, clarifying 
issues, and providing additional research information. 
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Following the first year of training, the McREL staff members become 
resource/support personnel who are responsive to directions and requests from 
the school teams. Building a long-term relationship between McREL staff and 
the teams to establish a continuous feedback and resource support system is 
an integral part of the McREL ESP. That system provides the Leadership Teams 
with a mechanism through which evaluation, renewal and revision activities 
can be developed and shared. Those activities are critical elements in 
successful long-term improvement efforts. 

Site Vist Synthesis 

During the 1984-85 school year, follow-up site visitations were made to 
the previous year's school districts involved in McREL's Effective School 
Program. As previously mentioned, site visits occur at the request of the 
Leadership Team. Few Leadership Teams from 1984-85 requested site visits 
during the Fall of 1985; most will request site visits in January, 1986. 
However, one district's teams did request a visit to their schools; a summary 
of that visit follows. 

Within the district seven schools were invited to participate in the 
McREL ESP. These schools had reported low achievement scores. Site visits 
to these schools were conducted in November, 1985. In most schools, the 
leadership team was engaged in involving the entire staff in the ESP effort. 
Through the use of various McREL ESP instruments, i.e., the Instructional 
Leadership Questionnaire and the Describing Your School's Characteristics 
questionnaire (DYSC) , Leadership Teams were able to determine areas of staff 
concern. One of the biggest changes described to the McREL visitors was the 
building of a "team effort" or comradery among faculties. The opinion was 
expressed that faculty members felt as though they had input into the school 
plans and that they could support the efforts undertaken. 

During the visit to one school within the district, the McREL visitor 
perceived a positive school climate. Student work was displayed in the 
hallways which were clear of furniture and equipment. The building was 
clean, pleasant and showed an academic focus by exhibiting academic awards. 
After the McREL visitor interviewed the leadership team it became evident 
that the team had concentrated on the following building goals for 1984-85 in 
keeping with their leadership plans. 

1. Have each classroom focus on five identified behavioral and academic 
expectations . 

2. Have all teachers collect observational data for student 
engagement . 

3. Have all teachers become aware of building academic efficiency and 
how they can posit ively affect efficiency through their own use of 
t ime in the ir classrooms . 

4. Have teachers make students aware of the importance of being in 
school to increase their classroom attendance rates. 

5. Have teachers improve the lines of communication with parents. 

6. Have teachers make improvements to ensure that students understand 
their assignments/work and in showing care and concern for 
students . 

7. Have teachers initiate classroom activities that promote school 
spirit and pride . 
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After meeting with the McREL visitor, the team agreed to implement the 
following goals and activities. 



1. Re-administer the Describing Your School's Characteristics (DYSC) 
instrument and analyze the results in light of the school goals for 



2. Re-examine and reorganize various school committees to work on the 
improvement of the school goals for 1985-86. 

3. Redefine and clarify the role of the team and the school's advisory 
team in assisting teachers in achieving the goals for 1985-86. 

In a visit to another district, the Leadership Team discussed their work 
in areas identified by the faculty and through ESP instruments. The team 
also provided faculty assisted inservice activities for the entire faculty. 
This provided very effective for building faculty morale and also promoted 
faculty ownership in improvement activities. 

Another Leadership Team within the district began including parents and 
community members in its school activities. A lack of p.arental involvement 
and a sense of community apathy had been apparent by the "poor attendance at 
meetings and support in bond issues. Using activities and research 
information provided by McREL, teachers began working together to reach the 
parents and the community. Several area businesses "adopted" the school and 
sponsored contests rewarding academic excellence among students. Parent 
visitations increased and meeting attendance improved. 

One school team, after identifying discipline as an area of concern, 
produced a faculty skit emphasizing appropriate and inappropriate behaviors 
for the students. Another team recognized parental support as an area of 
concern. Through the use of research materials provided during the training, 
concerted effort was made to reach more parents and involve them in the 
school. Attendance and enthusiasm grew at the parent/ teacher meetings 
because of the extra efforts made by the faculty. 

Coaching is one technique discussed during the ESP leadership training. 
This process helps teachers provide feedback to each other in a peer 
observation and support system. This technique was overwhelmingly adopted in 
several of the schools within the district, so much so that in one school, 
two-thirds of the faculty "is willingly participating." McREL staff provided 
written materials and videotapes to the school teams who are implementing 
coaching processes . 

Leadership teams are asked to document all of their activities so that 
they can assess growth, where they started and where they want to go. As a 
result of completing Year One of the McREL Effective Schools Program, one ESP 
district coordinator has documented all building accomplishments for 1984-85 
and made a listing of projected improvements for the 1985-86 school year. 

Data for the evaluation of Stage III: Level C was also collected using 
follow-up telephone interviews. There were two interview questions designed 
for this stage and level: 

o Staff communication and professional growth: 

Which of the following have improved since training? 
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75Z teacher/principal communication? 
753! teacher/principal cooperation? 
bOZ professional growth? 

o Principal behavior changes: 

Do you detect a positive change in your ovm 

behavior since training? Yes 90% No 10% 

In which of the following areas specifically? 

lOOZa nnounc ing and influencing 85Z reward 

90Z coaching 952 facilitating change 

|0T modeling 

Based on the information from the site visits and follow-up interviews, 
it appears as though the ESP does effect a change in school policies and 
practices at the building level. More specifically! Leadership Teams appear 
to be able to translate the information contained in the ESP training into 
action plans which they then implement at their building sites. 

STAGE IV: Level D 

The purpose of the Stage IV: Level D evaluation was to determine the 
effects of the McREL ESP within classrooms. The primary types of data 
collected for the 1984-85 ESP evaluation were site visit and follow-up 
interview data. The evaluation question for this stage and level was: 

Do staff members utilize the school improvement techniques designed to 
effect classroom variables? 

During the site visits it was observed that one Leadership Team had 
concentrated its efforts on increasing classroom instructional time^ and 
student engagement rates. Throughout the year via inservice sessions and 
faculty meetings the team continued to provide other teachers with 
information on and suggestions for accomplishing these goals. The leadership 
team reported a marked difference in their classes but had not collected 
objective data to support their observations. 

During the follow-up interviews » administrators were asked six questions 
directly related to classroom level implementation of ESP techniques. The 
results for those quest ions are reported below : 

o Teacher implementation of skills plus strategies. Do the 
teachers in your school use skills and strategies in their 
classrooms that are described by the ESP research? 

Yes 100% No 

Do teachers use various strategies to begin the school 

year? YeslOO% No 

Have these strategies changed as a result of training? 

Yes 95% No 5%_ 

o Teacher use of diagnostic and assessment techniques. 

Do teachers use evaluative measures sensitive to instruction? 

Yes 95%_ No _5% 
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Have teachers analyzed the curriculum to determine if critical 
content is included? Yes 90% No 10% 



o Principal leadership improvements. 

Have you improved your role as an instructional leader? 

Yes lOO% No 

Do you spend more time as an instructional leader since training? 

Yes B0% No 20% 
% of increase 19% 

o Teacher and principal use of effective instructional practices. 

Have there been increases in the following since training? 
70% precise instructional language? 
70% frequent classroom observations? 
85% cooperative development? 
80% sharing of materials? 

70% teacher/administrator cooperative training? * 

The follow-up interview suggested wide use of school improvement 
techniques at the classroom level. In fact, this was the area probably most 
commonly mentioned as changed as a result of the McREL ESP. 

Although there was no direct observational data to evaluate the effect 
at the classroom level of the 1984-85 McREL-ESP, there is evidence for this 
effect from previous McREL-ESP's (those programs conducted before 1984-85). 
Specifically, the Year End Report for McREL, Regional Research and Services 
Component (November, 1982) documents changes at the classroom level from 70 
classrooms. Within those classes the amount of time spent on the following 
non-academic activities was monitored prior to and after the McREL-ESP: 

-beginning management time 
-transition time 
-giving assignments 

-working with individuals while others wait 
-ending management activities 
-social activities 
-interrupt ions 

Using the AEI data collection technique described in Section II of this 
report it was found that participating teachers significantly decreased the 
time spent in non-academic activities ( from 10. 78% of the school day to 
7.33% of the school day). Also documented in the 1982 report are changes 
in: 

-teacher use of effective disciplinary techniques (e.g., consistent 

reinforcement of classroom rules) 
-classroom organizational techniques (e.g. , clarity of goals and 

assignments) 

-teacher use of effective motivational techniques (e.g. , provides 
learning activities that are varied) 
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Considering all information, the interview and site visit data from the 
1984-85 evaluation and data from previous years, it appears as though the 
McREL-ESP does effect classroom level change relative to the effective school 
variables . 

STAGE V: Level E 

The hypothesized effect of Stage V: Level E is on students. 
Specifically, it is assumed that at this stage and level the McREL-ESP will 
affect such factors as student engagement, success at various tasks, and 
mot ivat ion. 

The evaluation question for this stage and level was: 

Have the application of the ESP techniques caused improvements in 
student motivation, engagement and success? 

Again for the 1984-85 evaluation follow-up interview data was used. Within 
the interview two questions were designed specificaly for Stage IV: Level E. 
The results are reported below. 

Are techniques being used which improve student motivation? 

Yes 95% No 5%^ 

Do teachers communicate to students that they believe in them and 
expect them to be academicaly successful? 

Yes 100% No 

If one utilizes the data from previous ESP evaluations more objective 
evidence for Stage IV, Level E can be gathered. 

Again, using the 1982 report cited previously, it has been found that 
within 54 classrooms tested students' engagement rates increased 
significantly after the McREL-ESP (from .796 to .823). It was also found 
that students' success rates increased significantly (from .71 to .77). 

STAGE V: Level F 

At this stage and level the hypothesized effect is on student 
achievement as measured by standardized tests. The evaluation questions for 
this stage and level were: 

Has student achievement increased? 

Has inequity of effectiveness decreased? 

As indicated in Figure 3 the data collection technique for this level and 
stage is designated as "informal." This means that pieces of information 
gleaned from data relative to other levels and stages were used to obtain a 
sense of the ESP effect on their level and stage. For example, the principal 
interview contained three questions which indirectly got at the Stage V: 
Level F effect. The result for those questions are reported below: 

Do teachers use test results to plan instruction? 
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Yes 90% No 10% 

Do teachers use instructional techniques specifically designed to 
improve achievement? 

Yes 95% No 52^ 

Do you systematically analyze test scores by socio-economic status? 

Yes 5% No 952 

The most interesting response here is to the question about systematic 
analynis of the test scores by socio-economic level. Within the ESP 
participants are presented with a technique for analyzing achievement by SES. 
The technique clearly illustrates whether or not one or more groups within a 
school or district population continually underacheive . More often than not, 
when a school or district performs this analysis, they find that the low SES 
students are very probably not served by the system. The results for the 
survey question above would indicate that the importance of monitoring 
students by SES although strongly emphasized in the ESP is not internalized 
by participating schools and districts. However, the interview data suggests 
an increased awareness on the part of ESP participants of the need to monitor 
and stress success on standardized tests. 

As was the case for Stage V: Level E, if we included data from 
schools/districts which began the McREL ESP prior to the 1984-85 academic we 
obtain more objective data for Stage V: Level F. The data reported below comes from 
two sources. The first source is one of the sites trained in 1982-83. In Figure 6 
distribution of achievement scores by quartiles is reported for 1981-82 and 1983-84 
for grades 1, 3 and 5. 
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As indicated in Figure 6, none of the changes in test score distributions was 
statistically significant. However there was a strong trend across all grade 
levels toward increased achievement. That is, higher proportions of students 
were in the upper quartile distributions in 1983-8A than in 1981-82. 

The second site from which achievement data is reported also trained in 
1982-83. Comparative achievement data for 1981-82 vs 1983-8 reported below 
in two ways. In Figure 7 the grade equivalency scores for student the 
**localic percentile ranks" of 90» 75^ 50 and 25 are reported for the academ 
year priorthe to the training (1981-82) and the academic year immediatley 
after training the (1982-84). In Figure 8 are reported changed in 
distributions within first through the fourth quartiles using national 
percentle ranks. 



-33- 



Figure 7 



Comparision of National Grade Equivalency with 
Local Percentile Ranks by School Year 



Local 
Percentile 
Rank 


National Grade Equivalency of Blue Va 


ley Stud 


ents by 


Grade U 


'vel and Y 



ear 






90th 


3.67 


3.65 


-.02 


5'.67 


5.78 


+ .11 


8.15 


8.14 


-.01 


10.76 


11.72 


N.A. 




75th 


3.17 


3.25 


+ .08 


5.40 


5.43 


+ .03 


7.75 


7.75 




10.11 


10.98 


N.A. 




50th 


IM 


2.71 


+ .07 


4.90 


4.98 


+ .08 


7.13 


7.26 


+ .13 


9.29 


10.22 


N.A. 




25lh 


2.05 


2.21 


+ .16 


4.35 


4.41 


+ .06 


6.41 


6.52 


+ .11 


8.50 


9.58 


N.A. 






1981-82 


1983-84 


Change 


1981-82 


1983-84 


Change 


1981-82 


1983-84 


Change 


1981-82 


* 

1983-84 


Change 






S.D.».7 
N='238 


S.D.».7 
N«278 




S.D.".9 
N»233 


S.D.«.7 
N=273 


S.D.».9 
N=278 


S.D.=.9 
N»282 




S.D.=1.3 
N=247 


S.D.=1.1 
N=290 



Ist Grade 3rd Grade 5th Grade 7th Grade 

*8th Grade (1983-84) 



WINTER WORKS 



3/ 



3(] 



ERIC 



Figure 8 
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SECTION IV 

SUMMARY DISCUSSION 

Using a model of layered impact for the McREL ESP, six stages and levels 
were evaulated for the 1984->85 academic year. 

The hypothesized change at Stage II: Level A was in participants* 
knowledge of key elements of the school effectiveness literature. The 
comparative results of pre and post tests over this material indicated that 
part ici pants do increase their knowledge and under st and ing of this 
literature. To evaluate this stage and level workshop evaluation forms were 
also utilized. An analysis of these indicated very positive reactions to the 
ESP. Participants were especially positive about the content presented and 
the modeling of instructional techniques. 

At Stage II: Level B the expectations were that participants would begin 
to usei at their respective sites, the skills and strategies presented in the 
ESP. An analysis of team leadership plans and follow-up telephone interviews 
were used to evaluate this level and stage. The analysis of the leadership 
plans indicated an uneven effect. That is, there was a wide range of quality 
in the plans. However the telephone interviews indicated that in 80% of the 
schools surveyed the building leadership teams were actively engaged in 
further developing and implementing their leadership plans. 

The intent of Stage III: Level C was that building policies, practices 
and climate would experience a positive change. Here site visits and 
follow-up interviews were used. The site visits indicated that the McREL ESP 
hady indeed, impacted school policies, and practices and climates as was 
evidenced by increased colleagiality and mutual support among the entire 
staff for buildings participating in the ESP. The telephone interviews 
corrborated these findings. 

At Stage IV: level D the hypothesized effect of the ESP was at the 
classroom level. Specifically it was hypothesized that teachers would begin 
to use techniques which increase their efficiency in use of time managerial 
techniques, beginning the school etc. Results from the site visits and 
follow-up interviews indicated that the effect was strong for this level and 
stage. Participants do use the classroom level techniques presented in the 
ESP. They do so quite consciously and exhibit a noticeable and measureable 
shift in their classroom behavior. 

For Stage V: Level E. the intent of the ESP was for student behavior to 
change relative to engagement, motivation and success at academic tasks 
presented in class. Results of the follow-up interviews indicated that this 
does occur. There is a noticeable change in student behavior relative to 
these variables. If data from pre- 1984-85 ESP sites is also considered the 
behavioral change is qualifiable. Specifically, at pre -1984-85 rates it was 
found that students' engagement and success rates increased significantly 
after the McREL ESP. 

For the 1984-85 evaluation Stage V: Level F was evaluate informally 
since the hypothesion effect — student increase in academic achievemeni as 
measured by standardized tests — probably would not occur in the short 

-36- 



40 



Mount of tiM finct tht C8P was conducttd. Combintd retultt from tho 
foUoi#«ttp ifittrvitvt and titt vititt tutttsttd thtt in iht 1984-85 titet the 
**0t«gt it ttt** for incrtattd ttudtnt achitvtmtnt. If wt utt datt from pre 
-19I4-B5 tittt tht tvidtnct it ttrongtr for t Stage V: Ltvtl E effect. Some 
tittt txhibittd an incrtatt in over-all achievement on tttndardised tetlt. 
Nort itiportantly thit achitvewent it uniform acrott the quart ile 
dittribttt iont. That it, ttudentt at the lower qutrtilet of achievement 
increaae at leatt at much at ttudentt at the higher quart ilet. An apparent 
anonaljf at thit ttage and level it that ESP participantt do not appear to 
Miphatiaa tracking the achiavament of ttudentt by SES group even though thit 
it ttrettatf during the training. Although participantt all agree that thit 
typt of analytit it needed to inture equity in tervicet provided to ttudentt 
ftv participating building teamt or administratort ute the techniquet 
pretented them once the initial trainingt are completed. 

In tumary, for the tix ttaget and levelt evaluated for the 1984-85 
academic yeart evidence for hypothetited effectt vat found for all ttaget and 
levelt. Civen the methodical development of the McREL ESP thit it not 
turpriting. Over the yeart in which it hat been field teated componentt have 
been added and deleted to make it more efficient. The model that exittt now 
hat bean axtantively field tetted. One would then, expect the program to 
**delivar** on itt promitet. 

Thit doet not mean thtt the program it contidered finithed. At training 
continuet, feedback it conttantly tought tnd program changes made where 
appropriate. 
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BEST COPY AVAILABLE 

Mid-cone Lnenc Regional Sducacional Laboracory 
Ef fe.: ive Sohoo 1 s Program (£3?; 
Pre-?osc Workshop Quesc LonnaLre 

1. Whac is Che currenc research definicion of an effective school? 
An effective school is one chac — 



2. A great deal of research was conducted in classrooms during the 
70's and early 80's, che question chac was being a^ked was whac is 
effeccive inscruccion? The inosc pocencially useful variable co 
emerge from chac decade of research was . 

3. There has emerged from che ef fecc iveness research a powerful 
concepc called Academic Learning Time (ALT). ALT is composed of 
chree ways c ime is used in buildings and classrooms. They are 

, .1 and • 



4. In order for scudenc learning Co be mosc successful, chree 
elemencs of che curricul'om musc be aligned or in agreeraenc. The 
chree pares of che curriculum chac musc be aligned are 
, • anJ • 

5. Recenc educacion research indicaces chac che way a ceacher begins 
che school year is crucial Co scudenc success. Name four ouc of 
che eigne praccices effeccive ceachers apply during che firsc few 
weeks of school chac ineffective ceachers don'c apply. 

I. 
2. 
3. 
A. 

6. While obs'jrving che cimo-on-Cask of 3Cud.;nC:j during learning 
acciviti'is engaged raCi.s w.^re found Co be consiscencly che lowesc 
dar ing • 

7. liigh suc-.j-js r.ic.*s itic r^si.-i.-' sCud-Mic ach i.jv.*men: e:;pecially for low 
and middle acii i'W.* r s . Ta:jkj luv-? succes.-s rac.^s would be defined 

as chose ih-xt allow sC "uli-mc s co succood appro:< imac e ly % of 

Ch«i C imo . 

8. A ceaoher's value syscem i.s relat-id co scudenc achievemenC 
T^*aoh..»r5? may empliasi/.i^ many kinds of goals. Howover ceacher 

emphasis on' goals is posic ively associated 

wich scudenc ao. id-*mic learning. 

9. In.scruccion i.i ac cho h.Mrt of ceachlng, buC whac is ic chaL 
characcer izes effeccive inscruccion? Tliere is a vase body of 
knowledge? and re.se.ir.:h demonsc rac ing chac all effeccive ceachers 
incorporace cercain principles and phases or seeps co increase 
scudenc learning. Whac are chose phasi»s or seeps? Please lisc. 
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10. Teachers who exhibit Che highesc expeccacions for cheraselves and 

for cheir scudencs consider cheir primary role as ceachers Co be - 



11. Expeccacions for scudenc abilicy and performance are ofcen based 
on a single skill or faccor; chac skill is • 

12. If you synthesize che large body of research on expectations in 
the classroom there are three major areas where teachers treat 
students differently. They are: 

1. 
2. 
3. 

13. Motivation is generated by a person's needs, and is a force thac 
defines that person's actions. Research in scudent motivation has 
identified four interrelated factors that influence student 
motivation. These four factors are: 

1. 
2. 
3. 
4. 

14. The reality is chac mocivacion is an inCernal phenomenon. This 
means that nobody can mot ivace anyone else to do anything. Do you 
agree or disagree with chis scacemenc? 



15. As you mighc imagine, reiiearch has found cercain d is c i ngu ish ing 
feacures of schools wich eff*iccive discipline praccices. In 
gt^neral, five praccices are mencioned. Name chree of che five. 
I. 

2. 
3. 

16. There is evidence chac discipline problems and scudenc alienacion 
are rooced in che way schools organize ch.^mselves. Wliac school 
practices (ex. emphasis on competition) have you observed chac 
concribuce to alienation and discipline problems? 



17. T^^;ichers with tiu? b«.'s c -b*?l^.iv.»d class-fs pr.iVi?:it probl-imii by 

practicing six c la:J.s ri)i^m m.in.i;M'intMit techn i<i»je s . PU».is»? name 3 of 

thoso 6 techniques. 

I. 

t • 

3. 

13. Can you describe tho steps or pha:ii»s tliese instructional ''models" 

use to describe what effective teacliers do? 

1. Mad •» line Hiinter mod^l yes no 

2. Mastery Learnin»^ model yes no 

3. Barak Rosenshine's mod.?l yes no 

4. Missouri Mathematics Project yes no 

5. ' Fitzpatrick Secondary model yes no 

o ^ BEST COPY AVAILAB' 
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WORKSHOP TITLE t 



PAOTICI PANT'S EVALUATION FOKM 
Oir«ctlon»i complete thii form and return it to thf* worksjhor Leadf-r. 

Workshop Location I Pate ; 

Your achool or District i 

Position! 

Please indicate your reactions to the different aspects of the workshop, using 
the scales provided* Place a check ( ) on the line to show your rating. 

!• The workshop activities and presentationa l 

Excellent Good Pair Poor 

2* The workshop materials : 

Excellent Good Fair Poor 



3* Opportunities for participation : 

Excellent Good Fair Poor 

4* Opportunities for feed-back ; 

Excellent Good Fair Poor 

S* How the workshop was organized and scheduled over t)ie tine period provided: 
Excellent Good Fair Poor 

6* If you checked "Fair" or "Poor" for any of the preceeding areas, what 
could be done to improve these in another workshop? 

1. 

2* 

3. 
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7. What were the workshop's stongest contrabutauns to you, personally? 



1. 
2. 
3. 



fl. What do you feel were the workshop's greatest w eakne sses or limitations ? 
1. 



2. 
3. 



9. Rate the following workshop areas: Extremely Somewhat Not Very 

a. How valuable was the workshop as a 

source of useful information? 



b. How relevant was the information tc 
your organization? 

c. How useful was the workshop to you 
personally? 

10. Other comments or suggestions: 
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APPENDIX C 



ACADEMIC EFFICIENCY INDEX 
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ESTIMATING ACADEMIC EFFICIENCY 



The purpose of the accompanying worksheet is to develop an 
estimate of the percentage of the school day that students are engaged 
in academic work— -i.e., the efficiency with which time is used for 
instructional purposes. ("Academic work" is broadly defined to 
encompass any legitimate instructional area— reading, math, science, 
physical education, the arts, etc.) 

The process of arriving at an estimate of efficiency for a 
particular building can be visualized by looking at Figure 1. What 
you're doing, essentially, is making an estimate of the (I) Total Time 
for: (II) Non- instructional building activities (such as lunch, home 
room, etc.) , (III) Absenteeism, (IV) Non-instructional class activities 
(social activities, housekeeping tasks, etc.) , and (V) Student 
inattentiveness (the percentage of instructional time students aren't 
engaged in the work assigned to them) . Hie sum of the estimates for 
each of these blocks (II-V) , equals an estimate of^the "academic 
efficiency" of the building. 

Keep in mind that the figure is an estimate of academic 
efficiency. The intent is an index that approximates that true 
situation in the same way that an economic index, such as the "Gross 
National Product," estimates the health of the nation's productivity. 
As a result, the numbers emerging from the process should be viewed as 
only one source of information to be weighed against other data by 
people who know the situation first-hand. 

Note also that although we have found the academic efficiency of 
the engagement of students with relatively high agreement, the data 
should not be used to compare schools or teachers. !nie few 
observations made here would not be a reliable estimate of any given 
teacher's use of time though they are a reasonably valid estimate of 
time useage in the building as a whole. Comparison from one building 
to another is not appropriate because it has been our experience that 
each team involved in estimating these figures decide to define such 
things as non-scheduled academic building activities differently. For 
example, some may decide to include estimates for early dismissals and 
others do not. Therefore, the inability to generalize a teacher's use 
of time and the inconsistent definitions used between schools make the 
instrument inappropriate for purposes of comparison. 
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FIGURE 1 
ACADEMIC ErflCIENCY GRID 
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INSTF.UCTIONS FOR CLASSROOM OBSERVATIONS 



We suggest you make observations across several time periods — 
balance them between mornings and afternoon. If you are observing an 
elementary classroom start your observations midway through the 
transition from that activity to the mid-point of the next activity. 
(That way you pick up the time lost to transitions, etc.) If it's a 
secondary class be sure to start at the moment the bell rings. 

Before observing a class, identify the students. you want to 
observe. We require a minimum of nine students, but we suggest 
observing fifteen, the total number of student spaces on the Classroom 
Observation Vtorksheet: Side 2. The students you select to observe 
should be representative of the class. Thus you should select an equal 
number of high, medium, and low achievers. You will note that on the 
Classroom Observation worksheet: Side 2, there is space for each 
student's name and his/her achievement rating (H-M-L) . Every time you 
observe, you should follow the same students and rec ord their names and 
achievement codes . (If you are unfamiliar with students' names, you 
may want to add other identifying words — "the boy in the red shirt, - 
"the small girl with the curly hair," etc.) When you observe, be sure 
to fill in the following information in the blanks provided on the 
Classroom Observation Vtorksheet: Side 1: a) Date of Observation, b) 
the School, c) the Number of Students Absent, d) the Subject/Topic 
Class Observed, e) the Time Observation Began, f) Time Observation 
Ended, and g) the Total Time Observed. 

Once the preliminary information has been filled in on the 
worksheet, you can begin to observe classroom interactions. When 



observing the interaction in a classroom, you should be looking for two 
types of activities: 1) those activities during which instruction and 
learning are taking place, 2) those activities in which instruction and 
learning are not taking place. When instruction and learning are not 
occurring you use Side 1 of the Worksheet to record the type and amount 
of Time of non- instructional activity that is occurring. When 
instruction/learning is occurring, you use Side 2 of the Worksheet to 
record which students are engaged and which are not. 

To illustrate how this all works, assume you are observing a 
class. The bell rings and you note that no instruction/learning is 
taking place because the teacher is taking role call. You immediately 
begin timing this non-instructional activity using Side 1. When this 
activity ends, you record how long it took (e.g., 1 min. 20 sec. or_ 
1:20) in the box across from "Beginning Managerial Class Activities" 
(in column #1). As soon as instruction/learning begins, you shift your 
attention to Side 2 of the worksheet. You then begin to observe each 
of your selected students one at a time and determine whether or not 
that student is engaged in the teaching/learning activity s/he is 
supposed to be working on. Make sure you use the "snapshot in time 
technique" to make an accurate decision as to whether the student is 
focusing on his/her assigned task. If you decide the student is 
engaged, put a check in the student's row under the column for the 
first set of observations (column #1). If s/he is not engaged leave 
the box blank. You also have the option of using the coding sheet and 
entering the codes in the box blank. When you've made your decision 
for the first student, go on to the second student. Continue going 
from student to student until you've observed all selected students In 
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sequence. Repeat this sequence inunediately. Move to the next column 
under the Observation Cycles section after each complete cycle of 
engaged rate observations. You will note that there are places for 15 
separate observation sweeps; although during any one class you may not 
make 15 sets of observations. If r during the process of the 
observation r the teacher stops the class to deal with a disruptive 
studentr stop your observation, flip back to Side 1. (It's useful to 
have the observation sheets printed back-to-back) r and start timing the 
disruption. When it's over and you've entered the time involved and 
checked the appropriate activity, turn back to Side 2 and start 
observing the student you were last observing. If you have questions 
about the process, call for a "Trainer" in either the Kansas City or 
Denver office of McREL ~ (303)337-0990. 
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COMPARATIVE 


SUMMARY 


DATA 










ELEM. 


J.H. 


S.H. 


DISTICT 


1. 


Efficiency loss due to 
scheduling 


21 .1% 


17.8% 


16.6% 


19.2% 


2. 


Efficiency loss due to 
absenteeism 


4.1% 


4.1% 


8.0% 


4.8% 


3. 


Efficiency loss due to 
non-instructional class activities 


9.9% 


7.4% 


10.8% 


9.5% 


4. 


Efficiency loss due to 
student inattentiveness 


A J a. 

9.4% 


O. 4% 


/ . / 1 




5. 


Instructional efficiency 


55.6% 


64.4% 


58.0% 


57.7% 




Engagement rate 


85.1% 


90.6% 


88.4% 


87.1% 




Non-instructional class 
activities 


13.3% 


9.4% 


i:'.9% 


12.3% 




a) Beginning managerial 


1.1% 


1.8% 


3.4% 


1 .5% 




b) Transition time 


4.9% 


2.3% 


4.0% 


4.4% 




c) Teacher giving assignment 


3.7% 


1.4% 


2.3% 


3.2% 




d) Teacher disciplining 


.4% 


.3% 


.0% 


.3% 




e) Teacher working with 
1/2 students 


.4% 


.2% 


.5% 


.5% 




f) Ending managerial 


1.0% 


.9% 


1.0% 


1 .0% 




g) Social activities 


.0% 


.0% 


.9% 


.1% 




h) Interruptions 


.8% 


1.1% 


.6% 


.8% 
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CLASSROOM OBSERVATION WORKSHEET! SIDE 1 



a, DATE OP OBSERVATION 

b. SCHOOL 



C. NO. OF STUDENTS ABSENT 



e. TIME OBSERVATION BEGAN 

f. TIME OBSERVATION ENDED 

g. TOTAL TIME OBSERVED 



d. SUBJECT/TOPIC CLASS OBSERVED 



Instructions! tecord the total amount of Time (e.g., 1 minute 20 seconds or l!20) spent In each type of 
non-instructional activity In the box across from that activity. You will move from coIuei to column If the 
activity is repeated. Total each activity across the row to get the Total Time, then add the Total Time 
column to get the Total Time lost. 



0^ 

o 



Types of Non-Instructional Activities ; 
~"Beg Inning Managerial Class Activities 

(taking role, etc.) 
"Transition time (e.g., students getting 

out books, students passing In assignment) 
"Teacher giving the assignment 
"Teachei disciplining as student or the 

whole class-disrupts whole class 
"Teacher working with 1-2 students 

while others wait. 
"Ending Managerial Class Activities 

(students passing In books, etc.) 
--Social activities 
"Interruptions (e.g., announcements 

person enters room, etc.) 



Columns For Recording Periods of Non-Academic Activity 



f1 


n 


»3 


14 


15 


f6 


♦7 


«8 


19 






TOTAL 
TIME 






























































































































































1 




































TOTAL TIME LOST: 





to calculate Step Four (IV) on the "Academic Efficiency Worksheet" for this observation divide 

the Ttotal Time Lost by the Mil Time Observed.* Enter this figure In the appropriate 

line on Step IV on the Worksheet. 

*(Be sure to convert all minutes to seconds before you divide the Total Time lost by the Total Time Observed.) 
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CLASSROOM OBSERVATION WORKSHEET: SIDE 2 

Type of Activity (Recotd of 

minutes for 
each type) 

DATE OF OBSERVATION _ 

SCHOOL [ 

TEACHER 

TIME OBSERVATION BEGAN 



, TIHB OBSERVATION ENDED 



SUBJECT 

I of Students in Room _ 
{ of Students Enrolled 



SECTION 1 1 LEARNING OPPORTUNITIES USED 



Por each three minute obser 
place a check in the box if 
is engaged (e.g., box It] a 
blank If student is unengag 

NAME 


vation 
student 

nd leave 

ed. 
ACHVMT. 
RATING 
H/M/L 


' 1 

1 

1 


— 1 

1 

2 




— 1 

1 

4 


5 


6 


7 


6 


9 


10 


11 


12 


13 


14 


\i 


& 

n 

Record 
Sum of 
Engagmt. 


n 

»of 

OBS 


C 

Engaged 
Rate A/B 


STUDENT 11 








































STUDENT 12 








































STUDENT 13 








































STUDENT H 








































STUDENT #5 








































STUDENT 16 








































STUDENT 17 








































STUDENT 18 








































STUDENT 19 








































STUDENT 110 








































STUDENT 111 








































STUDENT 112 








































STUDENT 113 








































STUDENT 114 








































STUDENT t\S 








































TEACHER 








































6ri 


1 


CLA 
ENG 
RAT 
D/NO. ST 


TOTAL 
SSROOM 
AGEMENT 
E 

UDENTS 


D» 



Presentation of new information 
Review of old information 

Class Discussion 

Whole class instruction 
Small group instruction 
One small group, other 

independently 

All working independently ~~ 



CODING SHEET 



Student Activities 





Reading 


(R) 




Listening 


(L) 


Engaged 


Writing/Drawing 


(W) 


Student 


Discussing 


(D) 


Activities 


Taking test 


(T) 




Responding to questions 


(Q) 










Reading 


(NR) 




Listening 


(NL) 


Hon-engaged 


Writing/Drawing 


(W/D) 


Student 


No apparent activity 


(N) 


Activities 


Social interaction 


(S) 




Wandering 


(NW) 



Teacher Activities 

Instructional Activities: Non-instructional Activities 

I I Beginning Managerial Class 





Lecture 


(L) 


Activities (taking role, etc.) 




Using audio visuals 


(AV) 


Transition time (e.g., students 


Teacher 


Demonstration 


iO) 


getting out books r students 


Activities 


Monitoring 


(M) 


passing in assignment.) 


General 


Asking questions 


(Q) 


Teacher giving or explaining 




Listening 


(L) 


assignment. 




Socializing 


(S) 


Teacher disciplining as student or 



the whole class — disrupts whole 
class. 



Teacher working with 1-2 students 

while all others wait. 
Ending Managerial Class Activities 

(students passing in books, 

etc.) 
Social Activities. 
Interruptions (e.g. , 

announcements, person enters 

room, etc.) 
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ACADBHZC BPPZCZBNCy WORKSHEET 



I. Sttp Ontt tstimatt thd total timt Availablt in the school day. 

h. Subract tht tima stvidtnts arrivt at school from the time thay leave, 
using a 24-hour clock, (e.g., a 3t30 p.m. dismissal is 1Si30 on a 
24-hour clock.) State this total time available in minutes 



ZZ. Step Twot estimate the percentage lost to scheduled, non--acadereic 
building activities. 

B. Time scheduled (min.) for non-academic activities} 



1. lunch period 

2. homeroom 

3. breaks betMen classes 

4. recess 

5. other non-academic activities* 

6. Total (U2*f3*«*4*^S0 



C. Proportion of time scheduled outside of class is calculated by 
dividing line 6 in Step ^o by the Total Time placed in A of 
Step One. 

D. Proportion of time scheduled for class is found by subtracting C 
in Step Two from 1.00. (1.00 - C) « 

IZZ. Step Threes estimate of the average amount of absenteeism per day: 

B< Total students enrolled 

P. Average number of students absent per day 

C. Proportion of absenteeism (P-E) 

H. Proportion of students attending (1.00 - G) 



*Zf you want to be as precise as possible, this figure should include an average 
number of minutes lost per day for such things as inservice days, early 
dismissals, sports rallies, travel time to sports events, end-of-the-year 
ceremonies, assemblies, special celebrations, etc. 
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ZV« step FDurt •stimate of non-academic class activities: 



I. Average proportion of class time spent in non~academic activity 

(Follow instructions on the Classroom Observation Worksheet: Side 1) 

Teacher #1* *We suggest observing at least one 

Teacher #2 teacher per grade or department. 

Teacher #3 

Teacher #4 

Teacher #5 

Teacher #6 

Teacher #7 

Teacher #8 



TOTAL Divide this TOTAL by the number of teachers 

observed and enter here 



J. Proportion of the school day lost to in-class non-acaderoic 
activity (1 x D) 

Remainder of School Day (D - J) 

V« Step Five: estimate for inattentiveness 

L. Average engagement rate in each observed class: 
(Follow instructions for classroom observations*) 

Teacher #1 

Teacher #2 

Teacher #3 

Teacher #4 

Teacher #5 

Teacher #6 

Teacher #7 

Teacher #8 



TOTAL Divide this TOTAL by the number of teachers 

observed and enter here 



M. Proportion of school day engaged (L x K) « 

N. Average proportion of time lost in class to 
inattentiveness (1.00 - L) » 

0. Proportion of school day lost to 
inattentiveness (N x K) « 

1« Efficiency loss due to non-instructional 
building activities (C x H) « 

2. Efficiency loss due to absenteeism G « 

3. Efficiency loss due to non-instructional 
class activities (J x H) « 

Efficiency loss due to student 
inattentiveness (0 x H) « 

Academic efficiency (M x H) - 

64 7i) 
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APPENDIX D 
EXAMPLES OF DYSC PROFILE CHAKTS 
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EliEHL'NTARY SCHOOL A - 1982/B3 
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U 

(n 

w 0.50 

D 

< 

111 
> 
< 



0.25 



Ql 

1 2 3 4 5 6 7 8 9 10 11 B C D E 

IMPROVEMENT AREAS 

0 TEACHERS * ADMINISTRATORS 

7o 



ERIC 



fllDDU'; SCHOOL a - 19B2/fl3 
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Figure ^ 



ESP Training Session Evaluations* 
Detroit Public Scisools, 1984-85 



ACTIVITIES/ 
PRESENTATIONS 

MATERIALS 

PARTICIPATICW 

FEEDBACK 

ORGAN IZ*ATION 

USEFULNESS OF 
INFORMATION 

RELEVANCY FOR 
ORGANIZATION 

USEFULNESS 
PERSONALLY 

TOTAL 



Session 
I 



4.66 
4.78 
4.53 
4.59 
4.56 

5.00 

5.00 

4.86 
4.69 



Session 
II 



4.19 
4.50 
4.23 
4.23 
4.38 

4.56 

4.56 

4.56 

4.38 



Session 
III 



4.33 
4.71 
4.54 
4.54 
4.25 

4.94 

4.44 

4.67 
4.54 



Session 
IV 



4.67 
4.92 
4.58 
4.46 
4.50 

5.00 

4.79 

4.66 
4.69 



TOTAL 

4.47 
4.73 
4.47 
4.46 
4.43 

4.87 

4.68 

4.68 
4.58 



♦Excellent = 5; Poor = 1 
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Figure O 

ESP Training Session Evaluations* 
Emporia Public Schools, 1984-85 



ACTIVITIES/ 
PRESENTATIONS 

MATERIALS 

PARTICIPATION 

FEEDBACK 

ORGANIZATION 

USEFULNESS OF 
INFORMATION 

RELEVANCY FOR 
ORGANIZATION 

USEFULNESS 
PERSONALLY 

TOTAL 



Session 
I 



3.95 
4.25 
4.03 
4.13 
4.30 

4.20 

4.08 

4.13 
4. 13 



Session 
II 



3.92 
3.71 
4. 18 
4. 18 
3.84 

3,74 

3.49 

3.74 
3.83 



Session 
III 



3.80 
4.20 
3.83 
3.97 
3.73 

4.20 

4.13 

4.07 
3.99 



Session 

IV 



3.62 
4.05 
4.18 
4.05 
3.54 

3.97 

3.81 

3.81 
3.88 



TOTAL 

3.82 
4.05 
4.06 
4.06 
3.86 

4.02 

3.87 

3.93 
3.96 



♦Excellent ■ 5; Poor » 1 
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ESP Training Session Evaluations* 
Ft. Madison Public Schools, 1984*-85 



ACTIVITIES/ 
PRESEm'ATIGNS 

materia::^ 

participation 

feedback 

organization 

usefulness of 
information 

(v 

RELEVANCY FOR 
ORGANIZATION 

USEFULNESS 
PERSONALLY 

TOTAL 



Session 
I 



4.42 
4.33 
4.15 
4.09 
4.24 

4.58 

4.70 

4.45 
4.37 



Session 
II 



4.38 
4.53 
4.06 
4.21 
4.32 

4.76 

4.76 

4.65 
4.46 



Session 
III 



4.51 
4.71 
4. 14 
4.17 
4.43 

4.88 

4.60 

4.66 
4.51 



Session 
IV 



4.63 
4.66 
4.50 
4.38 
4.47 

4.92 

4.78 

4.93 
4.64 



TOTAL 

4.49 
4.56 
4.21 
4.21 
4.37 

4.78 

4.71 

4.66 
4.49 



♦Excellent = 5; Poor * 1 
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ESP Training Session Evaluations* 
Liberty Public Schools, 1984-85 



ACTIVITIES/ 
PRESEKTIATIONS 

MATERIALS 

PARTICIPATION 

FEEDBACK 

ORGANIZATION 

9 

USEFULNESS OF 
N INFORMATION 

RELEVANCY FOR 
ORGANIZATION 

USEFULNESS 
PERSONALLY 

TOTAL 



Session 
I 



4.44 
4.45 
3.72 
3.88 
4.19 

4.57 

4.45 

4.25 
4.24 



Session 
II 



4.15 
4.23 
3.87 
3.87 
4.11 

3.92 

3.81 

3.59 
3.94 



Session 
III 



4. 18 

4.:? 

4.08 
3.85 
4.08 

^..18 

3.76 

3.53 
3.99 



Session 
IV 



4.25 
4.43 
4.47 
4.14 
4.11 

4.06 

3.85 

3.53 
4.11 



TOTAL 

4.28 
4.34 
3.97 
3.92 
4. 13 

4.21 

3.99 

3.75 
4.07 



♦Excellent = 5; Poor = 1 
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Figure ^ 

ESP Training Session Evaluations* 
Sioux Palls Public Scnools, 1984-85 



ACTXVZTZES/ 
PRESEMTATZCMS 

fUTERZALS 

PARTZC. rZCN 

FEEDBACK 

ORGAN ZZ AT ZGN 

f 

USEFULNESS CP 
ZNPORMATZGN 

RELEVANCY FOR 
ORGAN ZZATZ ON 

USEFULNESS 
PERSCNALLY 

TOTAL 



Session 

Z 



4.29 

3.29 
3.43 
2.86 
3.57 

SkOO 

4.71 

4.71 
3.98 



Session 

ZZ 



4.14 
4.29 
4.14 

3.57 
3.57 

3.29 

3.29 

3.57 
3.73 



Session 
ZZZ 



4.56 
4.44 
3.89 
4.00 
4.44 

4.56 

4.33 

4.56 
4.35 



Session 
ZV 



5.44 
4.44 

4.11 
4.22 
3.89 

4.33 

4.56 

4.56 
4.49 



TOTAL 

4.66 
4.25 

3.90 
3.72 
3.90 

4.31 

4.25 

4.38 
4.17 



*£xcellent • 5; Poor - 1 
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Figure f 



ESP Training Session Evaluations* 
Winfield Public Schools, 1984-85 

Session Session Session Session 

I II III IV TOTAL 

ACTIVITIES/ 

PRESEMTATIGNS 4.1 1 4.00 4.30 3.85 4.06 

MATERIALS 4.41 4.35 4.19 4.23 4.29 

PARTICIPATION 3.96 3.61 3.58 4.35 3.87 

FEEDBACK 4.22 3.75 3.81 * 3.88 3.92 

ORGANIZATION 4.11 4.11 3.96 4.00 4.05 
USEFULNESS OF 

INFORMATION 4.33 4.14 4.20 4.44 4.28 
RELEVANCY FOR 

ORGANIZATION 4.15 .3.86 3.92 4.15 4.02 
USEFULNESS 

PERSONALLY 4.11 4.21 4.04 4.44 4.20 

TOTAL 4.18 4.00 4.00 4.17 4.00 

♦Vx eel lent ■ 5; Poor « 1 
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APPENDIX F 
STATED STRENGTHS AND WEAKNESSES OF ESP TRAININGS 
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Flgute J\ 



00 
0 



buestlom 

n, If f)\i checlietl 
ibU ot \wr fot any 
of tb^ pipcwilnq ar(»88, 
M could h» done to 
kiprove In 
ancthpr wrkiihop? 



7. What W5n the 
workshop's atrongest 
conttibutlona to you, 
petfionally? 



ESP Ttalninq Svaalon Evaluations 
Detroit Public SchooU, 19B4-85 

Strengths and Mtakneai of the Norkahop Beaalom 



Seaalon It 

Hotp diacuaaion/ 
aharlnq/lnteracticn |l| 



Information, materlala/ 
tools (1^1 

variety 

organltatlon 

presentation 

Presentees 15) 

Group interaction (2) 

Created awareness, 
need |1| 



Session Mi 

More discussion/ 
sharing/interaction |3| 

Vary niethod of 
presentation (1| 

■ Check for levels of 
understanding |)| 

' InfomiatioiVnaterlals/ 
txls |)B| 

teaching inodels 
' Presenters (3) 



Besslon IVt 

Don't schedule 
Saturday sessions 



Session Illi 

More dlBcuesloiV 
orlng/lnteractlon ()| 

v.ieck for levels of 
understanding (1) 



Infomatlon/naterials |9| - Inforaatlon/«'»teriali 1 18) 



> Opportunities for group 
interaction/planning |7| 

' Presenters (2) 



' Opportunities for 
interaction/dlKUsaion |6| 

' Presenters (51 

' local staff 12) 

' Contributing as a team 
nenUr (1) 



Reaffirined Ideas IM 
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ESP Training Session Evaluations 
Detroit Public Schooli, 1984-85 

Stcengths end Weakness of the Workshop Sessions 



8, What do you feel were 
the workshop's greatest 
weaknesses or linitations? 



- Too much information/ 
not enough time (4) 

- None (41 



Too much Infortnatlon/ 
not enough time (4| 

None HI 

' Repetition of handouts (U 
- Saturday session '(I) 



Too nuch information/ 
not enough tine (4) 

None 01 

Focus did not fit 
district goals 

' Saturday session (M 

• Consultants should work 
through academic efficiency 
chart before presenting (1) 



- Too iBuch inforwtion/ 
not enough tine (4) 

- None (4) 

- More contrast aiong 
presenters |)| 

- Working on school plan (1) 

- .Session should have 
b^en held earlier in 
the year |)| 

- Sessions 2 and 3 should 
have been reverses (1) 



8i) 



Question} 

6, If you checked 
fair or poor for any 
of the preceding areas, 
what toM be done to 
improve thesf in 
another workshop? 



figure 

BSP Trainlf^g Session Evaluations 
Emporia Public Schools, 19B4-BS 

Strengths ind Meakness of the Workshop SesElons 



Session I: 

' Less lecture/more 
discussion (4) 

' Hore breaks (tl 

' Eliminate smoking 



Session III 

- Unorganiied siaterialB 151 

* Confusion on vhat to do 
next (4 1 

- Too much information/ 
too little tine (2) 

- Less lecture, nore 
discussion HI ^ 

- Group size ID 

- Activities not applicable HI 

- Task analysis not applicable 



Session III: 

- Session too long (.'^ 

- Organization (31 

- Too mich inforiatioiV 
too little time (3| 

- Hore discussion/ 
hands-on activit.'.ei (2) 

»> 

- More specific toluLions (l| 



Session IVt 

- Of ganhat ion/prepara- 
tion 161 

- Too luch information/ 
too little time 111 

- Too ixH I ifne for 
planning :l) 



7. What were the 
wrkshop's strongest 
contributions to you, 
personally? 



Presenter/presenta- 
tion (81 

Organization 01 
Horklng with others |3| 
Reinforcement |2| 
' Enthusiasm (2| 
' Denionstration |)| 
' Location H) 



Infornatiori/uterials |29| ^ Infomiation/inaterialB (32) • Infonation/naterlals (30) 



research, 3 
instruction, 13 
handouts, I 
ESP, I 
seatwork, 2 
nanagenent, 1 
climate, 1 

teacher/administrator 
questionnaire |1| 

' Reinforcenent |3| 

- Interaction (3| 

' Planning (3| 

> Positive attitudes/ 
clarity (311 



discipline. It 
student lean) learning, 3 
motivation, 2 
expectations, 2 
STM), 1 ' 
ALT, 1 

effective strategies, 2 
Reinforcement (2| 
Sharing/discussion |2| 
Presenter (1| 



research, 5 
instruction, 3 
coaching, 1 
support groups, 1 

- Sharing, planning ()6| 
• Relnfcrcewnt (3) 

- Awareness (31 

- Presenter |l| 



- Food (or thought (2) 
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ESP Training Besiion Evaluations 
Bnporia Public Schools, 19eH5 

Streng; ^ ff^akneis of the VIbrUhop Sessions 



8» Hhat dc yoo feel were 
the wrjtksbop'g greatest 
weaknesses or limitations? 



Too much In format Ion/ 
too little time (P) 

lack of clarity- 
purpose and outccmes |)3| 

Sessions too long |4| 

> Location |3| 

' Directions needed 13) 

- Visuals (21 

• Too much lecture (2) 

• None (21 



- More directions/ 
suggestions (13) 

- Too much information/ 
too little time (6) 

- Too fast W 

- Hore participation |t| 

- Too long (1| 

' Survey forms (11 

- Handouts (1) 

^ Applicability (1) 



" Too much Infornation/ 
too little time (9) 

- Too much on Stident 
Teai Learning (3) 

- Hore specific exaiples (3| 

- Materials (2) 

- Sense of direction (l| 

- Too long (I) 

- Hare time for planning/ 
discussion W 

" location (1) 

- Lack of tiM for 
review (t| 

- Provide plans of 
action that have proven 
successful (t| 



- Too much information/ 
too little time [7 1 

- Hore direct loo/ 
examples (6) 

- Organization (4) 

- Eleoentary orientation (31 

- flelevancy (2| 

- Notebooks (2) 

- Too long (2| 

• Hasted tine (21 

- Facilities (t| 

- Kore time for sharing ()) 



1)J 



tlont 

6, If you checked 
fait ot poot for any 
of the preceding aread, 
vtiat could be doi.e to 
ijnprove theoe in 
anotliei wotkBhop? 



7, What wre the 
workshop's RtrongeRt 
contributions to you, 
personally? 



ESP Training Geasion Evaluations 
Ft. Madison Public Schools, 198H5 

Strengths and Weakness of the Morkshop Sessions 



Session It 

Too nuch inforioation/ 
too little tine (2) 

Hore time for presenters 
to meet with each tean (1 

More explanation of 
evaluation (1) 

' Use Bulti-itiedia 
approach |]| 

' Difficult script to 
read (1) 

• Classroom/leadership 
applications (IB) 

tine nansgenent, 10 
leadership, 2 



Session lit 

- Hurried presentation (2) 

* OrganlTation of 
materials (2) 

• ftore time for sharing/ 
questions (2| 

- Tine to n*et with 
presenters |)| i 



Session III: 

- Not enough time tc 
cover all topics |3) 

- Hore feedback (3) 

- Other examphs of 
fnotlvatlon (l| 

- Evenir)g session ()) 



Informatiorv^research (22) - Infornation/naterials (45| 



improving instruction, 6 
beginning the year, 4 
classroom nanageiDent, 3 



Research Information |)0| - Self and building 
assesflRient |6| 

ESP PtograiD |B| 



student tesp learning, 7 
discipline, B 
instructiorVteachlng, 4 
expectations, 4 
engageiKnt, 3 
motivation, 3 



organization 

objectives 

approach 

Information on instruc- 
tional effectiveness (7) 

Intetsction/Sharing (6) 

Motivation for 
itfiprcvenent |(| 

' Presenters (S) 

- Reinforcement of 
known concepts (3| 



Reinforcement of ideas (8| - Presenters ansi^rs, 
suggestions |2| 

Self fnotivation (6) 



fiesiiDP IVt 

Kore tlfff (21 

More follow-up (2) 

' Organization of 
materials (t) 



- Concrete imptovenent 
ideas IS) 

- Presenters (3| 

- Inter act lorv^tesRi 
building (3) 



- Reflection/Introspection (2) 

- Practical applications (2) 

- New Ideas (2) 

- Awareness of problems (1) 



Info nation/materials (20) 
Planning |7| 
Presenters (6) 
Support groups (6) 
(iioup interaction (4) 
Personal growth (3) 
Orod ekSffiples |U 
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ESP Tcalning Session EvaluUions 
Pt. Nadlion Public Echools, 1984-eS 

Strengths and tteakness of the Morkshop Sessions 



8. «iat do you feel «ece 
thf wckshop's 9ceatest 
veakAeises or Imitations? 



Too nuch Infornation/ 
too little tine (18) 



Too much infornation/ 
too little tiM (111 



• Need nee on "how to' (15) - 'How to'/pr act leal 

applications unclear (5) 

' Space too small |2| 

- Limited space (4) 

- Organliation (4) 



Too piuch inforiation/ 
too little tine (12) 

Hore *hov to* exaples |7) 

feedback |l| 

' Direction unclear (1) 

-Natedals distribution (l| 

' Assumption that there is 
tine to go back and do 
all this (1) 



bo little tlie (111 
Hone (4) 
Space (1) 
list ID 

toaeobly of iateri<l> (1) 



Sfiilons 2 atK) 3 should 
havt been reversed (1) 



Do 



Figure . P 
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Question: 

6. H you checlted 
fair or poor for any 
of the preceding areas, 
what could be done to 
Improve these in 
another workshop? 



7* Hhat vere the 
workBhop'e Btron9eBt 
contributions to you, 
personally? 



ESP Training Session Evaluations 
liberty Public Schools, I9fl<-B5 

Strengths and Makness of the Workshop Sessions 



Session 1 1 

- Group too large (31 

' Hore participation 
needed (31 

- Hore time (2| 

- Hore deinonstration 
and nodeling |2| 



Session lit 
- Materials (51 
' Group too large W 

• More tijue for questions/ 
discussion (<| 

• Overheads difficult 
to read (1| 



Session Hit 
> Group too large (2) 
• Shorter aessionsi:) 

- Hore breaks |2| 

- Better than session I! (1| 



Session IV: 

More planning tine (I) 

Too uordy (1) 

Better PA system (11 

> Condense coaching 
tape (1) 



- Clarlety of purpose (2) - Repetitions (1) 



- Overheads difficult 
to see 



• Hore sessions, less 
time per session (1| 

- Presentations too 
hurried (1) 

• TOO much reference to 
3rd session (1| 



Infonnatlon/materialB (57) - Hodels/technlquea (32) • Information/ideas (60) - InforttatioVwterials (25) 



engagement, 13 - Informatlon/wterials (13) 

school effectiveness, 6 
leadership^ 2 
handouts, 3 



instruction, 5 
success rateSf 3 

Presenter/r.eaenta- 

tion (IS) 

Positive attitudes/ 
expectations (M 

' Reinforcement (2) 

' Not yet ^1) 

^ Soul searching (1) 



' Reinforcement (7| 

• Planning process (41 

• Audio-visuals (2) 



- Motivation for 
Improvement (1| 



discipline, 13 
motivation, 12 
student ten learning, 12 
STM), 4 

s^lf-evaluation, 3 
engagement, 2 

' Sharing/discussion (21 
' Reneval (2| 



research, 3 
dlficlpUne, 2 
coaching, 2 
motivation, 1 
student team learning, 1 

* Sharing/discussion (23) 
- Reinforcement (<l 

• Presenters (1) 
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ESP Training Seaaion Bvaluatlona 
Liberty Public Schools, 198H5 

Strengths and WeaknesB of the (Workshop Sesaiona 



8. Hhat do you feel wre 
the workshop' I greatest 
veaknesaes or linitationa? 



00 



TOO Duch infornation/ 
Mo \itt\9 tin* 115) 


- Too Buch infornation/ 
too little tine (B| 


- Too nuch intonation/ 
too little tiae (7) 


* Too nuch infornation/ 
too little tine (4) 


Group size (11) 


- Group size (7) 


* Sessions too long (7| 


- Saturday seseions (3) 


Hore examplea/clari- 

flcation/flodeling (10) 


- Organization (i) 

- Sessions too long (2) 


* Group size (3) 

* Saturday sessions (2) 


- Sessions to long (1) 

- Group site 0) 


Sessions too long/ 
SatuMay aaaaions 


-Clarity/directions (1) 


- Poor visuals (1) 


- Hasted tiie (2) 


Overteaching, repiti- 
tion (4) 

Poor transparencies (3) 


- Poor visuals (1| 

• Too long between 
sessions (1) 


- Tine for breaks (1) 

- lack of clarity/ 
directions ID^ 


- Ibo long betwen 
sessions (2| 

* Not applicable to 
special areas (2) 


Not being paid (1) 
Other [i] 


- Not applicable to 
special areas (1) 

-Not being paid (1) 


- Markers (1) 


* Too little deaonstra- 
t ion/explanation (2) 



NO pay (I) 

Too much planning tine (I) 

* Not enough tin (or 
review (1) 

> Planning before goals 
are evaluated (1) 
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Figure £ 



OufBtion: 

6, If you checked 
fair or poor for any 
of the preceding areas, 
what could be done to 
Improve these In 
another workshop? 



ESP Training Session Evaluations 
Sioux Palli Public Schools, 1984-85 

Strengths and Weakness of the Workshop Sessione 
session I: Session lU Session III; 

- More tine (1) 

- Hore participation 



- Too much infonnation/ - Time schedule |1| 
too little time |6| 



More time for discus- 
sion |2| 



Session IV: 
More tine (1) 
Better organization (1) 



7, What wre the 
workshop's strongest 
contributions to you, 
personally? 



Assesslr^ time on task |5) - Materials/information (7) 
.Research awareness (3) • Discussion/feedback l2| 



' Haterials/infomation IS) - Mater iali/infomatlon (10) 



Opportunities for 
discussion (1) 

' Techniques for beginning 
change |1| 

> Evaluation of own 
activities |1) 



Administrator profiles |2| 
Explanation of DlfSC |l) 



discipline, 1 
motivation, 1 
student tem learning, 1 

Personal style 1 2) 

Review |2| 

New ideas l<) 

Interaction Ml 

Morale boosting 



coaching, 3 

teaching techniques, 1 
communication, 1 

- Working as a team )) 

- Presenters Ul 

• Growth opportunities (2) 

- Encouragement to tailor 
progran to needs PI 

■ Review |1| 

• Stress on relationship 
of activities to total 
concept (1) 



8. What do you feel were 
the workshop's greatest 
weaknesses or limitations? 



^ Too much Information/ 
too little time IS| 

- Not answering all 
questions |)| 

- Kissing the school 
day ID 



Too luch information/ 
too little time (3| 

Hore discussion needed |1| 

OrganUation 1 1) 

' Repetition (1) 



Too much information/ 
too little time (4) 



Ito much information/ 
too little time |S| 

Needed district 
uterials |1) 



Figure F 



Qu*6tiont 

6* If you checked 
fair or poor for any 
of the preceding areas, 
vhat could be done to 
iipro\'e these In 
another workshop? 



ESP Training Session Evaluations 
Nlnfield Public Schools, 19EiH3 

Strengths and Weakness of the Workshop Sessions 



Session It 

- Here opportunities for 
participation (3| 

- Less repetition |3| 

- Too much information/ 
loo little tine |3) 

- Need clarity of 
purpose (1) 



00 



Session lit 

- Too siuch lecture (6) 

" Too nuch information/ 
too little tine (4| 

* Too long (2) 

- Too slow (1| 

- Overhead8»diffic(tlt 
to read |l| 

" Getting group asslgn- 
inents done on tine (l| 



Session Hit 

Too nuch lecture |4| 

fiore opportunities for 
participation Ml 

Too much infornation/ 
too little tine |2| 

More direct questions (1) 

Inprove overheads |l| 

Need microphone |l| 



Session IV: 

More tine needed (I) 

More help planning (1) 

Hore directions/ 
expectations (t| 

Hore small-group 
activities Hi 

Meed nicrophone |1| 



7. rtiat were the 
wrkshop's strongest 
contributions to you, 
personally? 



Infornation, materials/ 
techniques (2I| 

Positive attitude toward 



Infornation/inaterials |2ei * Infornatlon/naterials (23| * Infornation/niBterlalB |I8| 



classcocn organization, 
t&anagenient, 5 



student team learning, 5 
expectations, 2 



Group interaction/ 
teaning |8| 



school inprovenent (9| 


instruction, S 


eifectlve teaching, 2 






node Is, 3 


motivation, 2 


- Relnforcenent (l| 


* Reinforcement (7| 




discipline, 2 






- Presenters |2| 


engagement rate, 1 


* Awireness of need (1) 


- Understanding program |7| 




team teaching, 1 






- Reinforcement |2| 


beginning school year, 1 


- Presenters (l| 


- Self evaluation |6| 










- Discussion (l| 


- Presenters |2| 


- Feedback |1| 


- Presenters |<| 










- Organization 


* Reinforcement (2| 


• Planning (]| 


* Lunch/goodies |2| 




* Discussion of plan (]| 


- Obiervatlons |l| 


- Tine for organitatlon |1| 














- Individual itylei ()| 


- Personal/professional 






•growth (1) 






- Self concept as an 
educator (1) 


- Haking nost of tine HI 
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tSf Tt lining S«stlon Evaluations 
Vlnfield Public Schools, I98<-B5 

Strengths and H^akntsa of the Norkshop Seasiona 



0 



H, rtiJl io pi tn\ wte 
the wrkahop'j greatest 
weaknesses or llnltations? 



' ''ouP Participation 
M*^ (91 

' *)!pl4nation 
M*^ HI 

' llniiatlon |4) 
' VlUon'(2| 

Jsti'"' (1) 



- Hore participation 


- Hore time for Interactlorv' 


- Too aiuch infonatlon/ 


needed (7) 


planning (7) 


too lltUe tile (9) 


- Too much Intormstlon/ 


- Poor aullo visuals |7| 


- Need clarlty-^oals/ 


too little tiifie |S| 




outcomes |4| 




films trips 




- Clarity of purpose/ 


overheads 


- More Intiraction 


outcane needed (4) 




needed (2) 




• Ibo much inforutlon/ 




*Nore specifics |3| 


too little tine |4| 


- Too luch tlie between 






sessions |2| 


- Revlev of Bession I (3| 


- Need clarity-goals/ 






expectations (2| 


" tare Infonation on 


- Too long (2| 




HcKEL (11 



Physical limitations (1| 
None 01 



- Too long (1| 

" Too much eiphasli on 
elementary (1| 

- Same preienter (1) 

- Hissed 1r. session (1) 

- Mo notebook till 
session HI (1| 

-N'^ne (1) 
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OnOER OQMOfIS ABOUT ESP TRAINING 
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Pigurt , 

BSP Training Stssion Evaluations 
Detroit Public Schools, 1984--8S 

Othtr CommeiiCS or Suggestions 



Ssssion Zt 

*Plsass givt Lynn a chanct to prtssnt without interruption.* 

"Have been to many similar types of programs - this is best organized 

a) written materials 

b) audio/visual 

c) oral presentation 

'Presenter prepared & knowledgable* 
Session ZXt 

organized and informative.* 
*X*m sure it will all sink in soon at a later date.* n 
Session Hit 

*Perhaps the total arrangement* of workshops could be different(?) - 
You* re experienced but - What about this: 1) as done 2) identification 
of needs via assessment instruments, DYSC, etc. 3) group planning of 
needs 4) then various folios & ideas according to identified needs.* 

*Should be a closer explanation and fit of McREL as it relates to the 
written School Improvement Plan.* 

*Thanks, Lynn and Barbara - Good Job! 

Session IV: 

*A Job well done.* 

*lleekend retreat approach may solve time crunch - relaxed.* 

*An exceedingly useful workshop with great potential for school 
improvement in the 7 participating schools as well as for potential 
replication in other schools of our district.* 

*Bandouts are fine mentioning concepts ^^re fine, but if they are to be 
remembered, used, etc., they need to be taught, not just mentioned. 
Perhaps a different match could be made, i.e., once a school knows its 
needs then relevant materials, info.« etc. could be shared according to 
needs, nianks for all your %#ork/ef fort.* 

*For Noble School - Workshops best after schools weekdays. Get us more 
aoney for staff development.* 
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ESP Training Session Evaluations 
Etoiporia Public Schools, 1984-85 

Other Coffljnents or Suggestions 

Session X: 

"I strongly believe in and support this process." 
"Very interesting." 

"Have handouts for every transparency." 

"Obviously-coaching strategy is quite important" 

"It was very well organized and representative did a great job." 

"I feel overwhelmed and need time to think about all of this." 

^ "Ms. Bverson is an interesting speaker, she presented the program to 
hold attention of the group." 

"Smoking only at break time." ^ 

"Prior knowledge was a plus for me-too many of my colleagues were 
not very aware of £ School movement techniques. Techniques are 
applicable to almost any area but especially adaptable to mine." 

"I don't feel holding the workshop at the country club is a good 
idea. Many people in the community feel that is a waste of 
taxpayer's money. This program is being lumped with the decision 
to hold the workshop at this location. I think this an unfortunate 
early assessment of ESP," 

"Please ask people to not smoke inside the room." 
"Limit smoking to breaks." 

"How much time will the project require of my time? My time for 
classroom teaching is limited already." 

"Getting into the Noteworthy ought to answer many of my questions. 
I still feel unsure about explaining this to others." 

"You're a very good speaker and really know your materials." 
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Emporia 
Page two 



Session II: 

"I think there are some expectations that the tearo will come back 
informed and trained to make sweeping changes in a short period of tiae," 

"I feel the workshop should be one week in stx:cession," 

"Confusion" 

"Conf used-not knowing where to go from here. We need a re':ipe!" 
"I guess I expect something to happen quicker," 

"I think that at least 2 board members should attend these sessions. I 
would like to hear the superintendent's views a'nd changing district 
philosophy." 

"Many thanks for the people not smoking in the meeting room. Mich 
better this time." 

Session III: 

"This was the best session of the three we've had, I feel." 
"Thank youl" 

"Keep up dessert & afternoon snacks." 

"Feel that much was presented. Still hard to absorb at once." 
Session IV: 

"It is too bad that the superintendent who pushed us into this program 
found that he was too busy to be present. The next person in charge came 
for lunch only. It is better to do as I do than as I say." 

"Does it strike you as slightly ironic that every school represented here 
had its administrators involved as part of each team EXCEPT the 
University?" 

"Susan & Lynn were extremely sensitive to the needs of our BLTs. The 
follow-up visits should also be productive." 

"I've enjoyed these sessions but it's hard to get through all the reading 
fit make good use of it." 

"Present t^e on peer conversations before observations start." 

"Good materials have been presented, nieir organization leaves a lot to 
be desired." 

"Enjoyed & appreciated the opportunity to join with the school district 
in this educational endeavor." 

"I would like to see the material organized in the notebook—an index or 
table of contents would be helpful." 
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ESP Training Session Evaluations 
Ft. Madison PuLdic Schools, 1984-85 

Other Comments or Suggestions 



Session I: 

"Can see possibilities - up to all of us to make it work" 
"Presentation - good" 

"I can better answer C later on into program" 

"I'll know a lot more when I actually use this material." 

"You really know your stuffM" 

"I wish we could have mentioned or partially covered classroom 
discipline." 

"Excellent presentation" 
"Great job, Susanl" 
Session II: 

"Thanksl" 

"It was very helpful to hear the methods and models discussed. Team 
discussion times helped to give us a good starti" 

"Good workshop. Valuable to me and the school district." 

"Super dayl Thanks" 

Session Ills 

"Wish we could have spent the whole day on discipline. I would like to 
know more about the assertive discipline method." 

"I feel these workshops are valuable." 

"Help in expanding to the rest of the faculty may be necessary." 
"Another very informative and stijnulating session" 
"An excellent job of presenting the material" 
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Port Madison 
Page two 

Session IV: 

"I enjoyed this workshop and learned from it." 
"This whole idea needs to not be dropped ." 
"Excellentl Happy Spring 1 

"Excellent presentation by a lady who was always prepared & organized." 
"Hope we continue to ure." 
"Super^ Susan!" 

"Amount of information was too much for amount of time available" 
"Collate paper be forehand 1" 

"Please collate papers in booklet at/or before first workshop." 
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Figure D 



ESP Training Session Evaluations 
Liberty Public Schools, 1984-85 

Other Comments or Suggestions 

Session I: 

"I think the info, should be cut dovm, presented in a variety of ways 
(videotapes, speakers in person), and more than 1 speaker should be 
used." 

■My building will not be participating at this time."" 

"A need to know that other schools go through some of the same 
difficulties" 



"If we hadn't been exposed to much of this before, the materials would 
have been overv*)elming. I don't feel this way, but I can understand how 
some people would." 

"Could some of the handouts be given out in advance so we would know what 
we were talking about before we zoomed through it." 

"Some things we hear repeatedly thru the year" 

"Doing these workshops would have helped to build a better attitude" 
"Good jobi" 

"Workshop will probably become more valuable as we apply it to our own 
class" 

"When will we ever have time to do all of this--we do not even have time 
to "disengage" our bladder during the teaching day." 

"Susan, you did a superb job in your presentation as always knowledgable 
and enthusiastic thank youl" 

"longer breaks (15 min.) too much sitting at a time" 

"This would be more effective if it were on district time instead of my 
timel" 



"Good to see the workshop-we already are implementing your sijggestions. 
It is very useful." 

"I know you were rushed; however, you need to cover questions so that 
understanding and clarification are possible." 

"Right now, I am worn out. Will this make things easier or harder?" 

"Need help in organization in time schedule and involvement of pupils 
vrnen 'seat work* seems to be necessary or alternative activity instead of 
seat work. Liked idea of coaching need enthusiasm. Entirely different 
from irtiat I anticipated" 

"Excellent presentation" 
Er|c 97 . , . 



Liberty 
Page two 

Session II: 

"I was enthusiastic about Session I and disappointed today." 

"This would be good to do a review session of right before the beginning 
of a new school year." 

"Fast paced r condensed, fairly easy to follow" 
"Most of this I learned in college." 

"In the future, I suggest eliminating December sessions — December 
Saturdays are too few and too precious. 

Session III: 

"Why do we have to keep filling out these evaluations?" 
"New materials presented this time" 
"I don't like filling these out." 

"These ideas sound "Pie in the Sky" when you realize our time structure*^ 
and stress on ou: district." 

"I don't like the evaluation at the end of each session." 
"Very interesting" 

"Felt more comfortable and not as threatened or offended as before - like 
gai7>e idea" 

Session IV: 

"I enjoyed the input." 

"Need another category between 'extremely' and 'somewhat' 
"Glad to be a part — the 2nd timel" 

"Why keep the answers a secret on the post-test. . .could be used for 
review" 

"Please use the post-test to review the main topics discussed. The 
answers are still a secret." 

"Why keep the answers to the pre- post-test a secret? I think doing the 
post-test as a group to review the most important aspects of McREL would 
be much more beneficial." 

"Not being a classroom teacher, the material for me was too ac-neral 
information. . .very interesting." 
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Figure E 



ESP Training Session Evaluations 
Sioux Falls Public Schools, 1984-85 

Other Comments or Suggestions 

Session I; 

■There was more active invclvement after lunch 

■Despite limitations, I'm particularly pleased that there is a method 
leading to total staff involvement rather than individual improvement 
goals and common ground for communicating hopes.^ 

Session II; 

■It is really difficult to sit for tnis length of time and to digest all 
the information. Two half-day sessions would be more helpful to me," 

"Perhaps having materials in advance would be helpful — at }.east for 
highly motivated people." 

Session III: k 

■This workshop was much better than Session II — better organized-more 
opportunity to break and discuss how the material applies to our 
situation. ■ 

Session IV: 

"I hope more schools will get involvtd in this worthwhile experience." 
■Looking forward to seeing you in 'e5-'86." 

"We have been presented with so much information that it is almost 
overwhelming. Maybe going through m^re of the information would have 
been helpful." 

"Appreciated the expertise of Sue & Lynn. Positive presentations, 
opportunities to choose ideas — not one rigid viewpoint. Feel we are not 
abandoned at the end of the workshops-support is there if needed." 

"I always enjoy these workshops and come away so inspired and challenged. 
Thanksl" 
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Figure F 

ESP Training Session Evaluation 
Winfield Public Schools, 1984-85 

Other Comments or Suggestions 



Session It 

"Excellent - this vrill be an outstanding opportunity to work together for 
a change r instead of struggling along wondering." 

"I enjoyed the day. Very useful information to me. I have a very 
positive attitude toward the final outcome." 

"I am looking forward to the next session." 

"I am roost anxious to see the analysis form NcREL develops on our tine 
utilization. Wish I had had this info when I will still in Reg. class 
room." 

«' 

■It's still early to integrate all that was presented today. In some 
ways I feel overwhelmed with v^at is expected of i[|e!" 

"Our teachers want to know how and why we were chosen for this." 

"Good presentation - I liked your style and enthusiasm. I an; excited 
about the program." 

■>tore time spent on how to implement program and less time on how program 
came about." 

"This early in the year, all new projects are rather overwhelming." 
"Well organized" 
"Good explanations" 
"Presented some good information" 
Session II: 

"We need more time here and at school to carry all this out." 
"Overall, information will be beneficial" 

"Need more specific information related to fields other than math and 
grammar" 

"I found this session less enticing than the 1st session, but I'm sure it 
will pick up." 
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Winfield 
Page two 



Session III: 

"Overall progress toward improving instruction in district is promising" 
"I am looking forward to usin9 many of the methods." 

"Today things are beginning to fit Into place and make more sense. The 
slower pace and concentration on specific areas are helpful.* 

"I think we H, school staffers have finally focused in on how to use this 
information we have been getting. 

Session IV: 

"Good presenter I" 

"I always enjoy learning more about my occupation and education in 
general. I feel it will make me a better person and my schools a better 
place to iearn." 

"I feel this is a good program. Is^hope we can do justice to it in our 
school and help make our school more effective and students DOre 
successful." 

"I hope I can convey the enthusiasm and knowledge of becoming more 
effective teachers to our building as your representative, Susan, did for 
me." 

"Questionnaire on this last day came when I was exceptionally tired and I 
could not think of answers which I knew previously." 

"Hope we can get the faculty excited about program." 

"School could allow time the next day for a follow-up and a chance to 
absorb what was covered the day before." 

"Thanks for coining 1." 
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